A Little More On Spondulas

As I mentioned before, Bart Hooper gave a great presentation on malware site analysis at Derby Con (suggest you watch the video if you monitor IDS and have to deal with end users accessing malicious sites). In his presentation he demo'd a tool he wrote called Spondulas. Spondulas is a web browser emulator and link parser. It grabs the raw output from the site, performs any needed post-processing, and saves an output file with the categorized links listed for you. Very nice tool that extends the functionality of tools like Malzilla.
 It's features (from the tool's Wiki site, found here) are:

• Support for GET and POST methods
• Parsing of retrieved pages to extract and categorize links
• Support for HTTP and HTTPS methods
• Support for non-standard port numbers
• Support for the submission of cookies
• Support for SOCKS5 proxy using TOR
• Support for pipe-lining (AJAX)
• Monitor mode to poll a website looking for changes in DNS or body content
• Input mode to parse local HTML files, e.g., e-mailed forms
• Automatic conversion of GZIP and Chunked encoding
• Automatic IP address Look-up
• Selection or generation of User Agent Strings
• Automatic creation of an investigation file

You can download either the Python-based  Linux version or the Windows version, which comes in 32 or 64 bit flavors. Excellent tool. Worth a second mention.