DirBuster -- Web Directory Enumeration
According to DirBuster's SourceForge, "DirBuster is a multi threaded java application designed to brute force directories and files names on web/application servers." You can also get the full technical discussion on the official OWASP DirBuster page.
Once you have DirBuster downloaded, simply launch it running the following: java -jar DirBuster_x.jar
You can also run it directly from the command line, without the GUI, as follows: java -jar DirBuster_x.jar -H -u https://target.com
The following GUI is DirBuster's front end. This is an example of my simple config:
Keeping your threads throttled is an important part of using DirBuster. Regardless of the threads you set, DirBuster is a noisy tool and will show up in the logs. But if your threads are set too high, you could noticeably effect the availability of the site, which could prompt the admins to check the logs! So it is advised to do some bench marking, and see what thread rate you can get away with without inducing errors (There are just more elegant ways to DoS). Using the auto-switching option will attempt to optimize the requests by using HEAD instead of GET if it can. This will cause smaller responses from the server if HEAD is successful otherwise it will cause a second request to be sent as GET, so this can be an important optimization feature. If the web server supports HEAD, you should probably use auto-switching.
An important part of using DirBuster is knowing whether or not your directories are case sensitive. This can be a quick test, by simply capitalizing a character in the directory part of your URL string. But what if you haven't been given access to a single directory yet? You can get the server's header information (curl -I [server_url]); if it's running Apache the directories are likely case sensitive, and if it's running IIS the directories are likely not case sensitive. This means you can run 'lowercase only' lists against IIS and not worry about missing anything while optimizing your scan. But for Apache you will have to run the larger, full case lists. Regardless, the best check is to always test the url string yourself, if you can. Below is a wordlist I chose for bruteforcing against Apache.
You will want to start simple, perhaps only bruteforcing directories at first, and not recursively until you know the areas and file types you want to more specifically target (My simple config is shown in the first DirBuster screenshot). Then it is best to go back and target areas more specifically, even using customized wordlists for known environments. If your making your own wordlist, the words are separated using a carriage return in a simple txt document.
Below is an example of DirBuster working successfully.
Notice that instantly certain directories present a wealth of hidden information such as /hidden/readme.txt, /hidden/source/, /js/, and /error/. The 'hidden' directory could hold old websites or even functionality the developers wanted to take out of production. The 'js' directory opens a wealth of modules and functions up to the attacker, and using the 'size' column the attacker can quickly identify the largest files. Lastly, the 'response' column shows that some directories are protected from the attacker with authentication controls such as /error/ and /cgi-bin/. These may be interesting to come back to later. Another useful tool is the 'Tree View', this allows you to see where your special files are clustered hierarchically.
Under 'options' -> 'advanced options' you can also up a username and password for authenticated directory scans using basic authentication. You can also customize your headers or run DirBuster through a proxy, such as a compromised machine, to obfuscate your true scanning location. Said proxy can also be set up with authentication as well. Fuzzing the URL can also help you manipulate the application to brute for hidden files.
Once you have DirBuster downloaded, simply launch it running the following: java -jar DirBuster_x.jar
You can also run it directly from the command line, without the GUI, as follows: java -jar DirBuster_x.jar -H -u https://target.com
The following GUI is DirBuster's front end. This is an example of my simple config:
Keeping your threads throttled is an important part of using DirBuster. Regardless of the threads you set, DirBuster is a noisy tool and will show up in the logs. But if your threads are set too high, you could noticeably effect the availability of the site, which could prompt the admins to check the logs! So it is advised to do some bench marking, and see what thread rate you can get away with without inducing errors (There are just more elegant ways to DoS). Using the auto-switching option will attempt to optimize the requests by using HEAD instead of GET if it can. This will cause smaller responses from the server if HEAD is successful otherwise it will cause a second request to be sent as GET, so this can be an important optimization feature. If the web server supports HEAD, you should probably use auto-switching.
An important part of using DirBuster is knowing whether or not your directories are case sensitive. This can be a quick test, by simply capitalizing a character in the directory part of your URL string. But what if you haven't been given access to a single directory yet? You can get the server's header information (curl -I [server_url]); if it's running Apache the directories are likely case sensitive, and if it's running IIS the directories are likely not case sensitive. This means you can run 'lowercase only' lists against IIS and not worry about missing anything while optimizing your scan. But for Apache you will have to run the larger, full case lists. Regardless, the best check is to always test the url string yourself, if you can. Below is a wordlist I chose for bruteforcing against Apache.
You will want to start simple, perhaps only bruteforcing directories at first, and not recursively until you know the areas and file types you want to more specifically target (My simple config is shown in the first DirBuster screenshot). Then it is best to go back and target areas more specifically, even using customized wordlists for known environments. If your making your own wordlist, the words are separated using a carriage return in a simple txt document.
Below is an example of DirBuster working successfully.
Notice that instantly certain directories present a wealth of hidden information such as /hidden/readme.txt, /hidden/source/, /js/, and /error/. The 'hidden' directory could hold old websites or even functionality the developers wanted to take out of production. The 'js' directory opens a wealth of modules and functions up to the attacker, and using the 'size' column the attacker can quickly identify the largest files. Lastly, the 'response' column shows that some directories are protected from the attacker with authentication controls such as /error/ and /cgi-bin/. These may be interesting to come back to later. Another useful tool is the 'Tree View', this allows you to see where your special files are clustered hierarchically.
Under 'options' -> 'advanced options' you can also up a username and password for authenticated directory scans using basic authentication. You can also customize your headers or run DirBuster through a proxy, such as a compromised machine, to obfuscate your true scanning location. Said proxy can also be set up with authentication as well. Fuzzing the URL can also help you manipulate the application to brute for hidden files.
