Looking at IR Tools
When performing incident response (most notably in a live situation), I find myself thinking, "There's gotta be an easier way." I've faced some of the very same situations you have...a multitude of systems that are physically/geographically distributed, but I can reach them via the network, Windows servers configured with no NetBIOS so you can't log into them remotely, etc. In all situations, however, the basic requirements are the same...the systems have to be examined live, can't be turned off, and you have to find out what, if anything, is going on. Basically, play "Where's Waldo" with malware.
The question has been...how best to do so? Well, I originally put together the FSP for this purpose. I wanted something that could be put together, was flexible and easy to use, and minimized the impact on the system. With the FSP, an investigator can put together a distribution CD, send it out to remote locations (or a first responder can download the files and burn them to CD), and the client will connect to the server over a TCP/IP connection to transfer the data that it collects.
What about other distribution methods, such as collecting information over the network? Well, if you're able to login remotely to Windows systems, you can use a combination of tools such as psexec.exe and WMI to collect information remotely. In fact, some of the tools I've created for use with the FSP use Perl to implement WMI.
Recently, I took a look at other tools that are available in this space. Let me start by saying a couple of things. First, what I'm going to say is based only on my initial impressions, not on extensive testing. When I began looking at these tools, I didn't have a set of criteria to evaluate them against. Rather, I wanted to get familiar with them, see how they were set up, how they functioned, etc. More extensive testing will come after I have a better understanding of the tools themselves. Part of the reason is that while the tools generally fit into the IR space, they are all different...different capabilities, functions, deployment options, etc.
Second, I do not make any claims that this is a complete list of the tools that are available. These are just the ones I'm aware of. If you find others, please don't hesitate to comment here, or drop me a line.
Finally, when I set out on this exercise, my notional testing infrastructure consisted of an XP Home SP2 laptop running VMWare Workstation 5.x and a Windows 2000 SP4 VM client. This gives me a semblence of a "network".
Now, let's take a look at the tools:
Mandiant's First Response - When I first downloaded First Response, I was concerned by the size...23+MB, plus an additional 22MB for the .NET framework. That's a lot of code, but it gets installed on one system, from which the agent files are distributed. Setting things up was relatively easy, and I was presented with a nice GUI (hey, I'm a fan of the command line, but I have no allusions that in order to penetrate the market, you NEED a GUI). I created the agent files with no trouble, and then copied them to a thumb drive. I had taken some time to puruse the Mandiant forums and seen how Steve Bunting had posted about an issue with distributing the FRAgent files. In a nutshell, the tool uses SSL to encrypt the data it sends, and creates individual keys for each system - so if you run the tools from a thumb drive on one system, you have to then "clean up" some of the files written to the thumb drive prior to running the agent on another system (from the thumb drive). This is inherent to the current version of First Response, and Dave Merkel indicated that this may change in the future. So, I copied the agent files to a directory on the Windows 2000 "guest" and attempted to install the agent (it installs as a Win32 service on the system...something to keep in mind); I say "attempted" because I received an error message about not being able to find a DLL that was clearly visible in the PATH.
Hhhhmmm. Okay, I then installed the agent files on the XP Home SP2 host system, and used the console to connect to the FRAgent running on 127.0.0.1. Things ran fine at that point, and I told the console to perform a "General Audit". After a bit, I could view all of the information that was collected...which was a lot. First Response grabs a pretty comprehensive set of information (Processes, Ports, Registry, file info, etc.) during a General Audit, and everything can be stored locally, and even exported to other formats (.cvs, .txt, etc.).
I think that First Response is best used if installed prior to an incident. The agent can be installed and running on servers, and if an incident occurs, the administrator can collect information from all of the systems, and then archive it and send it to someone for analysis.
Now, I should point out something very odd that I found. When I ran the General Audit on localhost, I noticed a connection in the Ports report...fragent.exe was connecting to an IP address off of my network, at nLayer/Akamai Technologies, via port 80. I posted this to the Mandiant forums, and will see if I can replicate this behaviour. At Matt Pepe's suggestion, if I see this again, I'll attempt to verify it with other tools, including WireShark.
Nigilant32 - Nigilant32 is a nice little tool from Agile Risk Management LLC...and I do mean little. It's a GUI tool, but still weighs in at under 1MB when you download the archive and unzip it. You can copy Nigilant32 to a thumb drive, and walk it around to various systems, launch the GUI, snapshot the system, save the retrieved data to the thumb drive, and move on.
The snapshot of data collected by Nigilant32 is comprehensive, but not easily parsed. Opening the snapshot in Notepad, you don't easily see all processes in relation to each other (due to how the information is formatted) and you have to go to another section of the file to see what ports are open.
Something nice about Nigilant32 is that you can dump the contents of RAM. I did that, and ran the dump through lsproc.pl and got the basic information about all the processes in RAM...very nice.
The one thing I didn't like about Nigilant32 is that when you launch the app, you're presented with a splashscreen. In my experience, the splashscreen should be visible as the app loads, or just for a couple of seconds. However, with Nigilant32, you have to click on the splashscreen to access the GUI. Annoying, but the Nigilant32 is still simple and easy to use, but one still needs some means of performing data reduction and correlation.
RPIER - This is an interesting little tool I came across from Intel, available on SourceForge. Unfortunately, I wasn't able to test it...I unzipped the archive and tried to run it, and the app complained that it couldn't find "mscoree.dll". I checked the archive, and didn't find any such file. I used Dependency Walker to verify that the EXE file did indeed rely on that DLL. The funny thing was, Mandiant's tool ships with a file by that name.
Looking at the files in the archive, it seems that RPIER is similar to the FSP, in that it uses and launches external tools to collect at least some of the information. Reading through the documentation on RPIER, it's a bit more mature than FSP, in that it has a GUI interface, comes pre-assembled, and the files can be uploaded to a properly configured web server, whereas the FSP doesn't write any files to the system being examined (it sends the information it collects out over a TCP/IP socket to the waiting server).
I've emailed one of the authors to see about resolving the issue...I'll test it out when the DLL issue is sorted out.
ProDiscover IR - This is the only non-free tool I looked at, and I'll tell you right up front...I've had a license for PD since around version 3.x. In fact, in the past year, I've been using it quite a bit, and it's my favorite tool for forensic analysis of images, by far. I've even gone so far as to convert EnCase files from their native format to dd-format so I can open them in ProDiscover.
PD/IR comes with an incident response capability...basically, an agent that can be either pushed out (via remote login) or distributed on CD, thumb drive, etc. Using the GUI to collect information can be complicated, but ProDiscover comes with the ProScript/Perl scripting language that allows you to automate sending the agent, collecting information, and then deleting the agent from the remote system. With multiple distribution methods and the ability to automate distribution and data collection, ProDiscover is the most flexible tool that I looked at.
Also, like Nigilant32, you can use ProDiscover IR to collect the contents of RAM from the remote system...only with PD, you can do it over the network.
In a nutshell, most of the tools I looked at have their uses and strong points. The one commonality amongst all of the tools is the lack of data reduction and correlation capabilities. However, that's not a bad thing...it's something that will come along as people use the tools, and as the tools mature. Collecting data is easy...analyzing the data is the hard part, and requires some skill and dedication. Like most incident responders, I've seen experienced administrators (and experienced responders, as well) look at the exact same data I'm looking at, and not see anything "unusual", or focus on the wrong thing simply because they aren't familiar with it. Data reduction and correlation tools can be crafted pretty easily...my preference and experience in doing so is with Perl...but at some point, a person needs to review the data and decide what's what. For example, if you're using any of these tools in a pretty static environment, such as a server farm or data center, you can pretty easily build a simple database (using mySql or SqlLite) of known-good processes, ports, etc. That way, after you run your collection tool(s), you can run the data through a parsing mechanism that filters out the known-good stuff...data reduction...leaving things that you need to look at (and may end up adding to the known-good list).
Again, please keep in mind that this "review" of the tools is just an initial blush, and I haven't explored all of the capabilities of each of them.
As always, comments/questions are welcome.
Addendum, 3 Aug: I was informed that I'd missed a tool that should have been included, WFT, which seems to have gained it's notariety through SANS. I downloaded it from the web site, and at first blush it's very similar to the FSP, in that you have to go out and get the tools you want to use with it. I wasn't able to easily locate anything in the zipped archive or at the web site that indicated where you can go to get some of the tools...some are obvious to me (such as those from SysInternals.com), while others aren't. There is one...mac.exe...that looks familiar (in name only), and may be one I wrote a while back, as according to the WFT configuration file, it was "compiled" with Perl2Exe.
I haven't run the tool yet, but from presentations and screenshots available on the web site, it looks like WFT reports it's output in a nice HTML format, in addition to saving the raw output of the commands that are run. Something like this is very useful, and is a step up from a simple batch file...WFT generates/verifies MD5 checksums of the tools listed in the configuration file.
All of these tools collect a good deal of information...some more than others. Tools such as WFT and FSP allow the user to configure how much information is collected (like the FSP, WFT is capable of using other configuration files besides the default), but the issue of data reduction, correlation, and analysis remains.
PSS: Okay, looking around, I've found other first response/IR toolkits; one at FIRST, and one called IRCR.
The question has been...how best to do so? Well, I originally put together the FSP for this purpose. I wanted something that could be put together, was flexible and easy to use, and minimized the impact on the system. With the FSP, an investigator can put together a distribution CD, send it out to remote locations (or a first responder can download the files and burn them to CD), and the client will connect to the server over a TCP/IP connection to transfer the data that it collects.
What about other distribution methods, such as collecting information over the network? Well, if you're able to login remotely to Windows systems, you can use a combination of tools such as psexec.exe and WMI to collect information remotely. In fact, some of the tools I've created for use with the FSP use Perl to implement WMI.
Recently, I took a look at other tools that are available in this space. Let me start by saying a couple of things. First, what I'm going to say is based only on my initial impressions, not on extensive testing. When I began looking at these tools, I didn't have a set of criteria to evaluate them against. Rather, I wanted to get familiar with them, see how they were set up, how they functioned, etc. More extensive testing will come after I have a better understanding of the tools themselves. Part of the reason is that while the tools generally fit into the IR space, they are all different...different capabilities, functions, deployment options, etc.
Second, I do not make any claims that this is a complete list of the tools that are available. These are just the ones I'm aware of. If you find others, please don't hesitate to comment here, or drop me a line.
Finally, when I set out on this exercise, my notional testing infrastructure consisted of an XP Home SP2 laptop running VMWare Workstation 5.x and a Windows 2000 SP4 VM client. This gives me a semblence of a "network".
Now, let's take a look at the tools:
Mandiant's First Response - When I first downloaded First Response, I was concerned by the size...23+MB, plus an additional 22MB for the .NET framework. That's a lot of code, but it gets installed on one system, from which the agent files are distributed. Setting things up was relatively easy, and I was presented with a nice GUI (hey, I'm a fan of the command line, but I have no allusions that in order to penetrate the market, you NEED a GUI). I created the agent files with no trouble, and then copied them to a thumb drive. I had taken some time to puruse the Mandiant forums and seen how Steve Bunting had posted about an issue with distributing the FRAgent files. In a nutshell, the tool uses SSL to encrypt the data it sends, and creates individual keys for each system - so if you run the tools from a thumb drive on one system, you have to then "clean up" some of the files written to the thumb drive prior to running the agent on another system (from the thumb drive). This is inherent to the current version of First Response, and Dave Merkel indicated that this may change in the future. So, I copied the agent files to a directory on the Windows 2000 "guest" and attempted to install the agent (it installs as a Win32 service on the system...something to keep in mind); I say "attempted" because I received an error message about not being able to find a DLL that was clearly visible in the PATH.
Hhhhmmm. Okay, I then installed the agent files on the XP Home SP2 host system, and used the console to connect to the FRAgent running on 127.0.0.1. Things ran fine at that point, and I told the console to perform a "General Audit". After a bit, I could view all of the information that was collected...which was a lot. First Response grabs a pretty comprehensive set of information (Processes, Ports, Registry, file info, etc.) during a General Audit, and everything can be stored locally, and even exported to other formats (.cvs, .txt, etc.).
I think that First Response is best used if installed prior to an incident. The agent can be installed and running on servers, and if an incident occurs, the administrator can collect information from all of the systems, and then archive it and send it to someone for analysis.
Now, I should point out something very odd that I found. When I ran the General Audit on localhost, I noticed a connection in the Ports report...fragent.exe was connecting to an IP address off of my network, at nLayer/Akamai Technologies, via port 80. I posted this to the Mandiant forums, and will see if I can replicate this behaviour. At Matt Pepe's suggestion, if I see this again, I'll attempt to verify it with other tools, including WireShark.
Nigilant32 - Nigilant32 is a nice little tool from Agile Risk Management LLC...and I do mean little. It's a GUI tool, but still weighs in at under 1MB when you download the archive and unzip it. You can copy Nigilant32 to a thumb drive, and walk it around to various systems, launch the GUI, snapshot the system, save the retrieved data to the thumb drive, and move on.
The snapshot of data collected by Nigilant32 is comprehensive, but not easily parsed. Opening the snapshot in Notepad, you don't easily see all processes in relation to each other (due to how the information is formatted) and you have to go to another section of the file to see what ports are open.
Something nice about Nigilant32 is that you can dump the contents of RAM. I did that, and ran the dump through lsproc.pl and got the basic information about all the processes in RAM...very nice.
The one thing I didn't like about Nigilant32 is that when you launch the app, you're presented with a splashscreen. In my experience, the splashscreen should be visible as the app loads, or just for a couple of seconds. However, with Nigilant32, you have to click on the splashscreen to access the GUI. Annoying, but the Nigilant32 is still simple and easy to use, but one still needs some means of performing data reduction and correlation.
RPIER - This is an interesting little tool I came across from Intel, available on SourceForge. Unfortunately, I wasn't able to test it...I unzipped the archive and tried to run it, and the app complained that it couldn't find "mscoree.dll". I checked the archive, and didn't find any such file. I used Dependency Walker to verify that the EXE file did indeed rely on that DLL. The funny thing was, Mandiant's tool ships with a file by that name.
Looking at the files in the archive, it seems that RPIER is similar to the FSP, in that it uses and launches external tools to collect at least some of the information. Reading through the documentation on RPIER, it's a bit more mature than FSP, in that it has a GUI interface, comes pre-assembled, and the files can be uploaded to a properly configured web server, whereas the FSP doesn't write any files to the system being examined (it sends the information it collects out over a TCP/IP socket to the waiting server).
I've emailed one of the authors to see about resolving the issue...I'll test it out when the DLL issue is sorted out.
ProDiscover IR - This is the only non-free tool I looked at, and I'll tell you right up front...I've had a license for PD since around version 3.x. In fact, in the past year, I've been using it quite a bit, and it's my favorite tool for forensic analysis of images, by far. I've even gone so far as to convert EnCase files from their native format to dd-format so I can open them in ProDiscover.
PD/IR comes with an incident response capability...basically, an agent that can be either pushed out (via remote login) or distributed on CD, thumb drive, etc. Using the GUI to collect information can be complicated, but ProDiscover comes with the ProScript/Perl scripting language that allows you to automate sending the agent, collecting information, and then deleting the agent from the remote system. With multiple distribution methods and the ability to automate distribution and data collection, ProDiscover is the most flexible tool that I looked at.
Also, like Nigilant32, you can use ProDiscover IR to collect the contents of RAM from the remote system...only with PD, you can do it over the network.
In a nutshell, most of the tools I looked at have their uses and strong points. The one commonality amongst all of the tools is the lack of data reduction and correlation capabilities. However, that's not a bad thing...it's something that will come along as people use the tools, and as the tools mature. Collecting data is easy...analyzing the data is the hard part, and requires some skill and dedication. Like most incident responders, I've seen experienced administrators (and experienced responders, as well) look at the exact same data I'm looking at, and not see anything "unusual", or focus on the wrong thing simply because they aren't familiar with it. Data reduction and correlation tools can be crafted pretty easily...my preference and experience in doing so is with Perl...but at some point, a person needs to review the data and decide what's what. For example, if you're using any of these tools in a pretty static environment, such as a server farm or data center, you can pretty easily build a simple database (using mySql or SqlLite) of known-good processes, ports, etc. That way, after you run your collection tool(s), you can run the data through a parsing mechanism that filters out the known-good stuff...data reduction...leaving things that you need to look at (and may end up adding to the known-good list).
Again, please keep in mind that this "review" of the tools is just an initial blush, and I haven't explored all of the capabilities of each of them.
As always, comments/questions are welcome.
Addendum, 3 Aug: I was informed that I'd missed a tool that should have been included, WFT, which seems to have gained it's notariety through SANS. I downloaded it from the web site, and at first blush it's very similar to the FSP, in that you have to go out and get the tools you want to use with it. I wasn't able to easily locate anything in the zipped archive or at the web site that indicated where you can go to get some of the tools...some are obvious to me (such as those from SysInternals.com), while others aren't. There is one...mac.exe...that looks familiar (in name only), and may be one I wrote a while back, as according to the WFT configuration file, it was "compiled" with Perl2Exe.
I haven't run the tool yet, but from presentations and screenshots available on the web site, it looks like WFT reports it's output in a nice HTML format, in addition to saving the raw output of the commands that are run. Something like this is very useful, and is a step up from a simple batch file...WFT generates/verifies MD5 checksums of the tools listed in the configuration file.
All of these tools collect a good deal of information...some more than others. Tools such as WFT and FSP allow the user to configure how much information is collected (like the FSP, WFT is capable of using other configuration files besides the default), but the issue of data reduction, correlation, and analysis remains.
PSS: Okay, looking around, I've found other first response/IR toolkits; one at FIRST, and one called IRCR.