Urlextractor - Data Gathering In Addition To Website Reconnaissance


Information gathering & website reconnaissance

Usage: ./extractor http://www.hackthissite.org/

Tips:
  • Colorex: seat colors to the ouput pip install colorex too purpose it similar ./extractor http://www.hackthissite.org/ | colorex -g "INFO" -r "ALERT"
  • Tldextract: is used past times dnsenumeration constituent pip install tldextract

Features:
  • IP too hosting information similar metropolis too terra firma (using FreegeoIP)
  • DNS servers (using dig)
  • ASN, Network range, Internet access provider cite (using RISwhois)
  • Load balancer test
  • Whois for abuse postal service (using Spamcop)
  • PAC (Proxy Auto Configuration) file
  • Compares hashes to diff code
  • robots.txt (recursively looking for hidden stuff)
  • Source code (looking for passwords too users)
  • External links (frames from other websites)
  • Directory FUZZ (like Dirbuster too Wfuzz - using Dirbuster) directory list)
  • URLvoid API - checks Google page rank, Alexa order too possible blacklists
  • Provides useful links at other websites to correlate amongst IP/ASN
  • Option to opened upward ALL results inwards browser at the end

Changelog to version 0.2.0:
  • [Fix] Changed GeoIP from freegeoip to ip-api
  • [Fix/Improvement] Remove duplicates from robots.txt
  • [Improvement] Better whois abuse contacts (abuse.net)
  • [Improvement] Top passwords collection added to sourcecode checking
  • [New feature] Firt run verification to install dependencies if need
  • [New feature] Log file
  • [New feature] Check for hostname on log file
  • [New feature] Check if hostname is listed on Spamaus Domain Blacklist
  • [New feature] Run a quick dnsenumeration amongst mutual server names

Changelog to version 0.1.9:
  • Abuse postal service using lynx istead of curl
  • Target server cite parsing fixed
  • More verbose near HTTP codes too directory discovery
  • MD5 collection for IP fixed
  • Links establish at in i lawsuit exhibit unique URLs from array
  • [New feature] Google results
  • [New feature] Bing IP banking concern gibe for other hosts/vhosts
  • [New feature] Opened ports from Shodan
  • [New feature] VirusTotal information near IP
  • [New feature] Alexa Rank information near $TARGET_HOST

Requirements:
Tested on Kali calorie-free mini AND OSX 10.11.3 amongst brew
sudo apt-get install bc coil dnsutils libxml2-utils whois md5sha1sum lynx openssl -y
Configuration file:
CURL_TIMEOUT=15 #timeout inwards --connect-timeout CURL_UA=Mozilla #user-agent (keep it simple) INTERNAL=NO #YES OR NO (show internal network info) URLVOID_KEY=your_API_key #using API from http://www.urlvoid.com/ FUZZ_LIMIT=10 #how many lines it volition read from fuzz file OPEN_TARGET_URLS=NO #open establish URLs at the terminate of script OPEN_EXTERNAL_LINKS=NO #open external links (frames) at the terminate of script FIRST_TIME=YES #if outset fourth dimension banking concern gibe for dependecies