Whatweb V0.5.0 - Side Past Times Side Generation Spider Web Scanner
Developed yesteryear Andrew Horton urbanadventurer together with Brendan Coles bcoles
Latest Release: v0.5.0. June 9th, 2019
License: GPLv2
This production is champaign of written report to the damage detailed inward the license agreement. For to a greater extent than information nearly WhatWeb visit:
Homepage: https://www.morningstarsecurity.com/research/whatweb
Wiki: https://github.com/urbanadventurer/WhatWeb/wiki/
If you lot convey whatever questions, comments or concerns regarding WhatWeb, delight consult the documentation prior to contacting 1 of the developers. Your feedback is e'er welcome.
About WhatWeb
WhatWeb identifies websites. Its finish is to answer the question, "What is that Website?". WhatWeb recognises spider web technologies including content management systems (CMS), blogging platforms, statistic/analytics packages, JavaScript libraries, spider web servers, together with embedded devices. WhatWeb has over 1800 plugins, each to recognise something different. WhatWeb every bit good identifies version numbers, e-mail addresses, describe of piece of job organisation human relationship IDs, spider web framework modules, SQL errors, together with more.
WhatWeb tin live stealthy together with fast, or thorough but slow. WhatWeb supports an aggression score to command the merchandise off betwixt speed together with reliability. When you lot watch a website inward your browser, the transaction includes many hints of what spider web technologies are powering that website. Sometimes a unmarried webpage watch contains plenty information to position a website but when it does not, WhatWeb tin interrogate the website further. The default score of aggression, called 'stealthy', is the fastest together with requires solely 1 HTTP asking of a website. This is suitable for scanning populace websites. More aggressive modes were developed for purpose inward penetration tests.
Most WhatWeb plugins are thorough together with recognise a make of cues from subtle to obvious. For example, most WordPress websites tin live identified yesteryear the meta HTML tag, e.g. '', but a minority of WordPress websites take away this identifying tag but this does non thwart WhatWeb. The WordPress WhatWeb plugin has over fifteen tests, which include checking the favicon, default installation files, login pages, together with checking for "/wp-content/" inside relative links.
Features
- Over 1800 plugins
- Control the merchandise off betwixt speed/stealth together with reliability
- Performance tuning. Control how many websites to scan concurrently.
- Multiple log formats: Brief (greppable), Verbose (human readable), XML, JSON, MagicTree, RubyObject, MongoDB, ElasticSearch, SQL.
- Proxy back upwards including TOR
- Custom HTTP headers
- Basic HTTP authentication
- Control over webpage redirection
- IP address ranges
- Fuzzy matching
- Result certainty awareness
- Custom plugins defined on the command line
- IDN (International Domain Name) support
Example Usage
Using WhatWeb on a twain of websites (standard WhatWeb output is inward colour):
$ ./whatweb slashdot.org reddit.com http://reddit.com [302] HTTPServer[AkamaiGHost], RedirectLocation[http://www.reddit.com/], Via-Proxy[1.1 bc1], IP[173.223.232.64], Akamai-Global-Host, Country[UNITED STATES][US] http://slashdot.org [200] Script, HTTPServer[Unix][Apache/1.3.42 (Unix) mod_perl/1.31], Google-Analytics[GA][32013], Via-Proxy[1.1 bc5], UncommonHeaders[x-fry,x-varnish,x-xrds-location,slash_log_data], Apache[1.3.42][mod_perl/1.31], HTML5, IP[216.34.181.45], OpenGraphProtocol[100000696822412], X-Powered-By[Slash 2.005001], Title[Slashdot: News for nerds, materials that matters], Email[canadaboy@nOspam.gmail.com,jbort@nww.com], Country[UNITED STATES][US] http://www.reddit.com/ [200] Frame, PasswordField[passwd,passwd2], Script, HTTPServer['; DROP TABLE servertypes; --], IP[203.97.86.202], JQuery, Cookies[reddit_first], Title[reddit: the vocalism of the network -- word earlier it happens], Country[NEW ZEALAND][NZ]Usage
.$$$ $. .$$$ $. $$$$ $$. .$$$ $$$ .$$$$$$. .$$$$$$$$$$. $$$$ $$. .$$$$$$$. .$$$$$$. $ $$ $$$ $ $$ $$$ $ $$$$$$. $$$$$ $$$$$$ $ $$ $$$ $ $$ $$ $ $$$$$$. $ `$ $$$ $ `$ $$$ $ `$ $$$ $$' $ `$ `$$ $ `$ $$$ $ `$ $ `$ $$$' $. $ $$$ $. $$$$$$ $. $$$$$$ `$ $. $ :' $. $ $$$ $. $$$$ $. $$$$$. $::$ . $$$ $::$ $$$ $::$ $$$ $::$ $::$ . $$$ $::$ $::$ $$$$ $;;$ $$$ $$$ $;;$ $$$ $;;$ $$$ $;;$ $;;$ $$$ $$$ $;;$ $;;$ $$$$ $$$$$$ $$$$$ $$$$ $$$ $$$$ $$$ $$$$ $$$$$$ $$$$$ $$$$$$$$$ $$$$$$$$$' WhatWeb - Next generation spider web scanner version 0.5.0. Developed yesteryear Andrew Horton (urbanadventurer) together with Brendan Coles (bcoles) Homepage: https://www.morningstarsecurity.com/research/whatweb Usage: whatweb [options] TARGET SELECTION: Enter URLs, hostn ames, IP adddresses, filenames or IP ranges inward CIDR, x.x.x-x, or x.x.x.x-x.x.x.x format. --input-file=FILE, -i Read targets from a file. You tin piping hostnames or URLs straight amongst -i /dev/stdin. TARGET MODIFICATION: --url-prefix Add a prefix to target URLs. --url-suffix Add a suffix to target URLs. --url-pattern Insert the targets into a URL. Requires --input-file, eg. www.example.com/%insert%/robots.txt AGGRESSION: The aggression score controls the trade-off betwixt speed/stealth together with reliability. --aggression, -a=LEVEL Set the aggression level. Default: 1. Aggression levels are: 1. Stealthy Makes 1 HTTP asking per target. Also follows redirects. 3. Aggressive If a score 1 plugin is matched, additional requests volition live made. 4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from all plugins are used for all URLs. HTTP OPTIONS: --user-agent, -U=AGENT Identify every bit AGENT instead of WhatWeb/0.5.0. --header, -H Add an HTTP header. eg "Foo:Bar". Specifying a default header volition supercede it. Specifying an empty value, eg. "User-Agent:" volition take away the header. --follow-redirect=WHEN Control when to follow redirects. WHEN may live `never', `http-only', `meta-only', `same-site', or `always'. Default: always. --max-redirects=NUM Maximum release of contiguous redirects. Default: 10. AUTHENTICATION: --user, -u= HTTP basic authentication. --cookie, -c=COOKIES Provide cookies, e.g. 'name=value; name2=value2'. --cookiejar=FILE Read cookies from a file. PROXY: --proxy Set proxy hostname together with port. Default: 8080. --proxy-user Set proxy user together with password. PLUGINS: --list-plugins, -l List all plugins. --info-plugins, -I=[SEARCH] List all plugins amongst detailed information. Optionally search amongst keywords inward a comma delimited list. --search-plugins=STRING Search plugins for a keyword. --plugins, -p=LIST Select plugins. LIST is a comma delimited laid of selected plugins. Default is all. Each chemical constituent tin live a directory, file or plugin get upwards together with tin optionally convey a modifier, eg. + or - Examples: +/tmp/moo.rb,+/tmp/foo.rb title,md5,+./plugins-disabled/ ./plugins-disabled,-md5 -p + is a shortcut for -p +plugins-disabled. --grep, -g=STRING|REGEXP Search for STRING or a Regular Expression. Shows solely the results that match. Examples: --grep "hello" --grep "/he[l]*o/" --custom-plugin=DEFINITION\tDefine a custom plugin named Custom-Plugin, --custom-plugin=DEFINITION Define a custom plugin named Custom-Plugin, Examples: ":text=>'powered yesteryear abc'" ":version=>/powered[ ]?by ab[0-9]/" ":ghdb=>'intitle:abc \"powered yesteryear abc\"'" ":md5=>'8666257030b94d3bdb46e05945f60b42'" --dorks=PLUGIN List Google dorks for the selected plugin. OUTPUT: --verbose, -v Verbose output includes plugin descriptions. Use twice for debugging. --colour,--color=WHEN command whe ther color is used. WHEN may live `never', `always', or `auto'. --quiet, -q Do non display brief logging to STDOUT. --no-errors Suppress mistake messages. LOGGING: --log-brief=FILE Log brief, one-line output. --log-verbose=FILE Log verbose output. --log-errors=FILE Log errors. --log-xml=FILE Log XML format. --log-json=FILE Log JSON format. --log-sql=FILE Log SQL INSERT statements. --log-sql-create=FILE Create SQL database tables. --log-json-verbose=FILE Log JSON Verbose format. --log-magictree=FILE Log MagicTree XML format. --log-object=FILE Log Ruby object inspection format. --log-mongo-database Name of the MongoDB database. --log-mongo-collection Name of the MongoDB collection. Default: whatweb. --log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0. --log-m ongo-username MongoDB username. Default: nil. --log-mongo-password MongoDB password. Default: nil. --log-elastic-index Name of the index to shop results. Default: whatweb --log-elastic-host Host:port of the elastic http interface. Default: 127.0.0.1:9200 PERFORMANCE & STABILITY: --max-threads, -t Number of simultaneous threads. Default: 25. --open-timeout Time inward seconds. Default: 15. --read-timeout Time inward seconds. Default: 30. --wait=SECONDS Wait SECONDS betwixt connections. This is useful when using a unmarried thread. HELP & MISCELLANEOUS: --short-help Short usage help. --help, -h Complete usage help. --debug Raise errors inward plugins. --version Display version information. (WhatWeb 0.5.0). EXAMPLE USAGE: * Scan example.com. ./whatweb seek out ple.com * Scan reddit.com slashdot.org amongst verbose plugin descriptions. ./whatweb -v reddit.com slashdot.org * An aggressive scan of wired.com detects the exact version of WordPress. ./whatweb -a three www.wired.com * Scan the local network speedily together with suppress errors. whatweb --no-errors 192.168.0.0/24 * Scan the local network for https websites. whatweb --no-errors --url-prefix https:// 192.168.0.0/24 * Scan for crossdomain policies inward the Alexa Top 1000. ./whatweb -i plugin-development/alexa-top-100.txt \ --url-suffix /crossdomain.xml -p crossdomain_xml Logging & Output
The next types of logging are supported:
- --log-brief=FILE Brief, one-line, greppable format
- --log-verbose=FILE Verbose
- --log-xml=FILE XML format. XSL stylesheet is provided
- --log-json=FILE JSON format
- --log-json-verbose=FILE JSON verbose format
- --log-magictree=FILE MagicTree XML format
- --log-object=FILE Ruby object inspection format
- --log-mongo-database Name of the MongoDB database
- --log-mongo-collection Name of the MongoDB collection. Default: whatweb
- --log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0
- --log-mongo-username MongoDB username. Default: nil
- --log-mongo-password MongoDB password. Default: nil
- --log-elastic-index Name of the index to shop results. Default: whatweb
- --log-elastic-host Host:port of the elastic http interface. Default: 127.0.0.1:9200
- --log-errors=FILE Log errors. This is commonly printed to the shroud inward red.
Plugins
Matches are made with:
- Text strings (case sensitive)
- Regular expressions
- Google Hack Database queries (limited laid of keywords)
- MD5 hashes
- URL recognition
- HTML tag patterns
- Custom ruby code for passive together with aggressive operations
$ ./whatweb -lWhatWeb Plugin List
Plugin Name - Description -------------------------------------------------------------------------------- 1024-CMS - 1024 is 1 of a few CMS's leading the agency amongst the implementation... 360-Web-Manager - 360-Web-Manager 3COM-NBX - 3COM NBX telephone system. The NBX NetSet utility is a spider web interface i... 3dcart - 3dcart - The 3dcart Shopping Cart Software is a consummate ecommerce s... 4D - 4D spider web application deployment server 4images - 4images is a powerful web-based picture gallery management system. Fe... ... (truncated)Search Plugins
To persuasion to a greater extent than exceptional nearly a plugin or search plugins for a keyword:
$ ./whatweb -I phpBB WhatWeb Detailed Plugin List Searching for phpBB ================================================================================ Plugin: phpBB -------------------------------------------------------------------------------- Description: phpBB is a costless forum Website: http://phpbb.org/ Author: Andrew Horton Version: 0.3 Features: [Yes] Pattern Matching (7) [Yes] Version detection from blueprint matching [Yes] Function for passive matches [Yes] Function for aggressive matches [Yes] Google Dorks (1) Google Dorks: [1] "Powered yesteryear phpBB" ================================================================================Plugin Selection
All plugins are loaded yesteryear default.
Plugins tin live selected yesteryear directories, files or plugin names every bit a comma delimited listing amongst the -p or --plugin command describe option.
Each listing item may convey a modifier: + adds to the total set, - removes from the total laid together with no modifier overrides the defaults.
Examples
- --plugins +plugins-disabled,-foobar
- --plugins +/tmp/moo.rb
- --plugins foobar (only take foobar)
- -p title,md5,+./plugins-disabled/
- -p ./plugins-disabled,-md5
The --grep, -g command describe selection searches the target page for the selected string together with returns a fit inward a plugin called Grep if it is found.
Aggression
WhatWeb features several levels of aggression. By default the aggression score is laid to 1 (stealthy) which sends a unmarried HTTP GET asking together with every bit good follows redirects.
--aggression, -a 1. Stealthy Makes 1 HTTP asking per target. Also follows redirects. 2. Unused 3. Aggressive Can brand a handful of HTTP requests per target. This triggers aggressive plugins for targets solely when those plugins are identified amongst a score 1 asking first. 4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from all plugins are used for all URLs.An illustration of the dissimilar results betwixt score 1 together with score 3:
Influenza A virus subtype H5N1 score 1, stealthy scan identifes that smartor.is-root.com/forum/ uses phpBB version 2:
$ ./whatweb smartor.is-root.com/forum/ http://smartor.is-root.com/forum/ [200] PasswordField[password], HTTPServer[Apache/2.2.15], PoweredBy[phpBB], Apache[2.2.15], IP[88.198.177.36], phpBB[2], PHP[5.2.13], X-Powered-By[PHP/5.2.13], Cookies[phpbb2mysql_data,phpbb2mysql_sid], Title[Smartors Mods Forums - Reloaded], Country[GERMANY][DE]$ ./whatweb -p plugins/phpbb.rb -a three smartor.is-root.com/forum/ http://smartor.is-root.com/forum/ [200] phpBB[2,>2.0.20]WhatWeb has no caching therefore if you lot purpose aggressive plugins on redirecting URLs you lot may fetch the same files multiple times.
Performance & Stability
WhatWeb features several options to increase performance together with stability.
- --max-threads, -t Number of simultaneous threads. Default: 25.
- --open-timeout Time inward seconds. Default: 15
- --read-timeout Time inward seconds. Default: 30
- --wait=SECONDS Wait SECONDS betwixt connections This is useful when using a unmarried thread.
Changing the user-agent using the -U or --user-agent command describe selection volition avoid the Snort IDS dominion for WhatWeb.
If you lot are scanning ranges of IP addresses, it is much to a greater extent than efficient to purpose a port scanner similar massscan to discovery which convey port fourscore opened upwards earlier scanning amongst WhatWeb.
Character laid detection, amongst the Charset plugin dramatically decreases performance yesteryear requiring to a greater extent than CPU. This is required yesteryear JSON together with MongoDB logging.
Optional Dependencies
To enable MongoDB logging install the mongo gem. jewel install mongo
To enable grapheme laid detection together with MongoDB logging install the rchardet gem. jewel install rchardet cp plugins-disabled/charset.rb my-plugins/
Writing Plugins
Plugins are piece of cake to write. Start yesteryear going through the plugin tutorials inward the my-plugins/ folder.
After progressing through the tutorials read through the Development department of the wiki.
Updates & Additional Information
The WhatWeb evolution construct features regular updates.
- Check the evolution branches for unreleased updates.
Release History
- Version 0.5.0 Released June 9th, 2019
- Version 0.4.9 Released Nov 23rd, 2017
- Version 0.4.8-dev (Continuous unloosen from 2012 to 2017)
- Version 0.4.7 Released Apr 5th, 2011
- Version 0.4.6 Released March 25th, 2011
- Version 0.4.5 Released August 17th, 2010
- Version 0.4.4 Released June 29th, 2010
- Version 0.4.3 Released May 24th, 2010
- Version 0.4.2 Released Apr 30th, 2010
- Version 0.4.1 Released Apr 28th, 2010
- Version 0.4 Released March 14th, 2010
- Version 0.3 Released at Kiwicon III (kiwicon.org), Nov 2nd, 2009
Credits
Developers
- Andrew Horton (@urbanadventurer)
- Brendan Coles (@bcoles)
Contributors
Thank you lot to the next people who convey contributed to WhatWeb.
- Emilio Casbas
- Louis Nyffenegger
- Patrik Wallström (@pawal)
- Caleb Anderson (@dirtyfilthy)
- Tonmoy Saikia
- Aung Khant (@yehgdotnet)
- Erik Inge Bolsø
- nk@dsigned.gr
- Steve Milner (@ashcrow)
- Michal Ambroz
- Gremwell
- Sagar Prakash Junnarkar (@sagarjunnarkar)
- GertBerger
- Quintin Poirier
- Eric Sesterhenn
- dengjw (@jawa)
- Pedro Worcel (@droop)
- Matthieu Keller (@maggick)
- Peter (2pvdl)
- Napz (@RootCon)
- @nilx042
- Fabian Affolter (@fabaff)
- Andrew Silvernail (@buff3r)
- Andre Ricardo (@andrericardo)
- nikosk
- Patrick Thomas (@coffeetocode)
- Guillaume Delcaour (@guikcd)
- Sean (@wiifm69)
- Matthieu Keller (@maggick)
- Raul (@raurodse)
- Andrew Petro (@apetro)
- Artem Taranyuk (@610)
- Matti Paksula (@matti)
- Tim Smith (@tas50)
- Sarthak Munshi (@saru95)
- @rdubourguais
- @SlivTaMere
- @Code0x58
- @iGeek098
- @andreas-becker
- @csalazar
- @golewski
- @Allactaga
- @lins05
- @eliasdorneles
- @sigit
- dewanto
- @elcodigok
- @SlivTaMere
- @anozoozian
- Bhavin Senjaliya (@bhavin1223)
- Janosch Maier (@Phylu)
- @rmaksimov
- Naglis Jonaitis (@naglis)
- Igor Rzegocki (@ajgon)
