Hostintel - A Modular Python Application To Collect Tidings For Malicious Hosts

This tool is used to collect diverse intelligence sources for hosts. Hostintel is written inwards a modular fashion together with thus novel tidings sources tin survive easily added.
Hosts are identified past times FQDN host name, Domain, or IP address. This tool exclusively supports IPv4 at the moment. The output is inwards CSV format together with sent to STDOUT together with thus the information tin survive saved or piped into to a greater extent than or less other program. Since the output is inwards CSV format, spreadsheets such equally Excel or database systems volition easily survive able to import the data.
I created a brusque introduction for this tool on YouTube: https://youtu.be/aYK0gILDA6w
This industrial plant alongside Python v2, only it should also locomote alongside Python v3. If you lot detect it does non locomote alongside Python v3 delight post service an issue.

Help Screen:

$ python hostintel.py -h usage: hostintel.py [-h] [-a] [-d] [-v] [-p] [-s] [-c] [-t] [-o] [-i] [-r]                     ConfigurationFile InputFile  Modular application to expect upward host tidings information. Outputs CSV to STDOUT. This application volition non output information until it has finished all of the input.  positional arguments:   ConfigurationFile     Configuration file   InputFile             Input file, 1 host per trace of piece of job (IP, domain, or FQDN                         host name)  optional arguments:   -h, --help            exhibit this assist message together with leave of absence   -a, --all             Perform All Lookups.   -d, --dns             DNS Lookup.   -v, --virustotal      VirusTotal Lookup.   -p, --passivetotal    PassiveTotal Lookup.   -s, --shodan          Shodan Lookup.   -c, --censys          Censys Lookup.   -t, --threatcrowd     ThreatCrowd Lookup.   -o, --otx             OTX past times AlienVault Lookup.   -i, --isc             Internet Storm Center DShield Lookup.   -r, --carriagereturn  Use wagon returns alongside novel lines on csv.

Install:
First, brand certain your configuration file is right for your computer/installation. Add your API keys together with usernames equally appropriate inwards the configuration file. Python together with Pip are required to run this tool. There are modules that must survive installed from GitHub, together with thus survive certain the git ascendance is available from your ascendance line. Git is slowly to install for whatever platform. Next, install the python requirements (run this each fourth dimension you lot git clit this repository too):

$ pip install -r requirements.txt

There accept been to a greater extent than or less problems alongside the stock version of Python on Mac OSX (http://stackoverflow.com/questions/31649390/python-requests-ssl-handshake-failure). You may accept to install the safety division of the requests library alongside the next command:

$ pip install requests[security]

Lastly, I am a fan of virtualenv for Python. To brand a customized local installation of Python to run this tool, I recommend you lot read: http://docs.python-guide.org/en/latest/dev/virtualenvs/

Running:

$ python hostintel.py myconfigfile.conf myhosts.txt -a > myoutput.csv

You should survive able to import myoutput.csv into whatever database or spreadsheet program.
Note that depending on your network, your API fundamental limits, together with the information you lot are searching for, this script tin run for a real long time! Use each module sparingly! In furnish for the long wait, you lot salve yourself from having to clit this information manually.

Sample Data:
There is to a greater extent than or less sample information inwards the "sampledata" directory. The IPs, domains, together with hosts were picked at random together with past times no way is meant to target whatever arrangement or individual. Running this tool on the sample information industrial plant inwards the next way:

Small Hosts List:

$ python hostintel.py local/config.conf sampledata/smalllist.txt -a > sampledata/smalllist.csv *** Processing 8.8.8.8 *** *** Processing 8.8.4.4 *** *** Processing 192.168.1.1 *** *** Processing 10.0.0.1 *** *** Processing google.com *** *** Processing 212.227.247.242 *** *** Writing Output ***

Larger Hosts List:

$ python hostintel.py local/config.conf sampledata/largerlist.txt -a > sampledata/largerlist.csv *** Processing 114.34.84.13 *** *** Processing 116.102.34.212 *** *** Processing 118.75.180.168 *** *** Processing 123.195.184.13 *** *** Processing 14.110.216.236 *** *** Processing 14.173.147.69 *** *** Processing 14.181.192.151 *** *** Processing 146.120.11.66 *** *** Processing 163.172.149.131 ***  ...  *** Processing 54.239.26.180 *** *** Processing 62.141.39.155 *** *** Processing 71.6.135.131 *** *** Processing 72.30.2.74 *** *** Processing 74.125.34.101 *** *** Processing 83.31.179.71 *** *** Processing 85.25.217.155 *** *** Processing 93.174.93.94 *** *** Writing Output ***

Intelligence Sources:
You tin larn API keys at the sites below for your configuration file.

• GeoLite2 (No network I/O required)
  • http://www.maxmind.com
• DNS (Network I/O required)
  • https://github.com/rthalley/dnspython
• VirusTotal (Public API fundamental together with network I/O required, throttled when appropriate)
  • http://www.virustotal.com
• PassiveTotal (API key, username, together with network I/O required)
  • http://www.passivetotal.com
• Shodan (API fundamental together with network I/O required)
  • http://www.shodan.io
• Censys (API key, username, together with network I/O required)
  • http://www.censys.io
• ThreatCrowd (Network I/O required, throttled when appropriate)
  • http://www.threatcrowd.org
• OTX past times AlienVault (API fundamental together with network I/O required)
  • https://otx.alienvault.com
• Internet Storm Center (Network I/O required)
  • https://isc.sans.edu

Resources:

• The GeoIP2 Python library
  • https://github.com/maxmind/GeoIP2-python
• The Python DNS library
  • https://github.com/rthalley/dnspython
  • Foundation of DNS lookups inspired past times http://www.iodigitalsec.com/performing-dns-queries-python/
• The VirusTotal Python library
  • https://github.com/blacktop/virustotal-api
• The Shodan Python library
  • http://shodan.readthedocs.io/en/latest/
  • https://github.com/achillean/shodan-python
• The Censys Python library
  • https://github.com/censys/censys-python
  • https://www.censys.io/api
• The PassiveTotal Python library
  • https://passivetotal.readthedocs.io/en/latest/
  • https://github.com/passivetotal/python_api
• The ThreatCrowd Python library
  • https://github.com/threatcrowd/ApiV2
  • https://github.com/jheise/threatcrowd_api
• The OTX Python Library
  • https://github.com/AlienVault-Labs/OTX-Python-SDK
  • https://otx.alienvault.com/api/
• The Internet Storm Center DShield Python Library
  • https://github.com/rshipp/python-dshield
  • https://isc.sans.edu/api/

Notes:
Crude notes are available here.

Download Hostintel