This post is about actually detecting persistence mechanisms...not querying them, but detecting them.  There's a difference between querying known persistence mechanisms, and detecting previously unknown persistence mechanisms used by malware; the former we can do with tools such as AutoRuns and RegRipper, but the latter requires a bit more work.

Detecting the persistence mechanism used by malware can be a critical component of an investigation; for one, it helps us determine the window of compromise, or how long it's been since the system was infected (or compromised).  For PCI exams in particular, this is important because many organizations know approximately how many credit card transactions they process on a daily or weekly basis, and combining this information with the window of compromise can help them estimate their exposure.  If malware infects a system in a user context but does not escalate it's privileges, then it will mostly likely start back up after a reboot only after that user logs back into the system.  If the system is rebooted and another user logs in (or in the case of a server, no user logs in...), then the malware will remain dormant.

Detecting Persistence Mechanisms
Most often, we can determine a malware persistence mechanism by querying the system with tools such as those mentioned previously in this post.  However, neither of these tools is comprehensive enough to cover other possible persistence mechanisms, and as such, we need to seek other processes or methods of analysis and detection.

One process that I've found to be very useful is timeline analysis.  Timelines provide us with context and an increased relative confidence in our data, and depending upon which data we include in our timeline, an unparalleled level of granularity.

Several years ago, I determined the existence of a variant of W32/Crimea on a system (used in online banking fraud) by creating a timeline of system activity.  I had started by reviewing the AV logs from the installed application, and then moved on to scanning the image (mounted as a volume) with several licensed commercial AV scanners, none of which located any malware.  I finally used an AV scanner called "a-squared" (now apparently owned by Emsisoft), and it found a malicious DLL.  Using that DLL name as a pivot point within my timeline, I saw that relatively close to the creation date of the malicious DLL, the file C:\Windows\system32\imm32.dll was modified; parsing the file with a PE analysis tool, I could see that the PE Import Table had been modified to point to the malicious DLL.  The persistence mechanism employed by the malware was to 'attach' to a DLL that is loaded by user processes that interact with the keyboard, in particular web browsers.  It appeared to be a keystroke logger that was only interested in information entered into form fields in web pages for online banking sites.

Interestingly enough, this particular malware was very well designed, in that it did not write the information it collected to a file on the system.  Instead, it immediately sent the information off of the system to a waiting server, and the only artifacts that we could find of that communication were web server responses embedded in the pagefile.

Something else to consider is the DLL Search Order "issue", often referred to as hijacking. This has been discussed at length, and likely still remains an issue because it's not so much a specific vulnerability that can be patched or fixed, but more a matter of functionality provided by the architecture of the operating system.

In the case of ntshrui.dll (discussed here by Nick Harbour, while he was still with Mandiant), this is how it worked...ntshrui.dll is listed in the Windows Registry as an approved shell extension for Windows Explorer.  In the Registry, many of the approved shell extensions have explicit paths listed...that is, the value is C:\Windows\system32\some_dll.dll, and Windows knows to go load that file.  Other shell extensions, however, are listed with implicit paths; that is, only the name of the DLL is provided, and when the executable (explorer.exe) loads, it has to go search for that DLL.  In the case of ntshrui.dll, the legitimate copy of the DLL is located in the system32 folder, but another file of the same name had been created in the C:\Windows folder, right next to the explorer.exe file.  As explorer.exe starts searching for the DLL in it's own directory, it happily loaded the malicious DLL without any sort of checking, and therefore, no errors were thrown.

Around the time that Nick was writing up his blog post, I'd run across a Windows 2003 system that had been compromised, and fortunately for me, the sysadmins had a policy for a bit more extensive logging enabled on systems.  As I was examining the timeline, starting from the most recent events to occur, I marveled at how the additional logging really added a great deal of granularity to thing such as a user logging in; I could see where the system assigned a token to the user, and then transferred the security context of the login to that user.  I then saw a number of DLLs being accessed (that is, their last accessed times were modified) from the system32 folder...and then I saw one (ntshrui.dll) from the C:\Windows folder.  This stood out to me as strange, particularly when I ran a search across the timeline for that file name, and found another file of the same name in the system32 folder.  I began researching the issue, and was able to determine that the persistence mechanism of the malware was indeed the use of the DLL search order "vulnerability".

Creating Timelines
Several years ago, I was asked to write a Perl script that would list all Registry keys within a hive file, along with their LastWrite times, in bodyfile format.  Seeing the utility of this information, I also wrote a version that would output to TLN format, for inclusion in the timelines I create and use for analysis.  This allows for significant information that I might not otherwise see to be included in the timeline; once suspicious activity has been found, or a pivot point located, finding unusual Registry keys (such as those beneath the CLSID subkey) can lead to identification of a persistence mechanism.

Additional levels of granularity can be achieved in timelines through the incorporation of intelligence into the tools used to create timelines, something that I started adding to RegRipper with the release of version 2.8. One of the drawbacks to timelines is that they will show the creation, last accessed, and last modification times of files, but not incorporate any sort of information regarding the contents of that file into the timeline.  For example, a timeline will show a file with a ".tmp" extension in the user's Temp folder, but little beyond that; incorporating additional functionality for accessing such files would allow us to include intelligence from previous analyses into our parsing routines, and hence, into our timelines.  As such, we may want to generate an alert for that ".tmp" file, specifically if the binary contents indicate that it is an executable file, or a PDF, or some other form of document.

Another example of how this functionality can be incorporated into timelines and assist us in detecting persistence mechanisms might be to add grep() statements to RegRipper plugins that parse file paths from values.  For example, your timeline would include the LastWrite time for a user's Run key as an event, but because the values for this key are not maintained in any MRU order, there's really nothing else to add.  However, if your experience were to show that file paths that include "AppData", "Application Data", or "Temp" might be suspicious, why not add checks for these to the RegRipper plugin, and generate an alert if one is found?  Would you normally expect to see a program being automatically launched from the user's "Temporary Internet Files" folder, or is that something that you'd like to be alerted on.  The same sort of thing applies to values listed in the InProcServer keys beneath the CLSID key in the Software hive.

Adding this alerting functionality to tools that parse data into timeline formats can significantly increase the level of granularity in our timelines, and help us to detect previously unknown persistence mechanisms.

Resources
Mandiant: Malware Persistence without the Windows Registry
Mandiant: What the fxsst?
jIIR: Finding Malware like Iron Man
jIIR: Tracking down persistence mechanisms

Many times we'll come across a case where we need to determine the presence of malware on a system.  As many of us are aware, AV products don't always work the way we hope they would...they don't provide us with 100% coverage and detect everything that could possibly affect our systems.

This post is NOT about malware analysis.  This post addresses malware detection during dead box analysis.  Malware detection is pretty expansive, so to really address the topic, I'm going to spread this one out across several blog posts.

Malware Detection
Malware detection during dead box analysis can be really easy, or it can be very hard.  I say this because we can mount an image as a read-only volume and run several (or more) AV scanners against the volume, and keep track of all the malware found.  Or, we can run several AV scanners against the volume, and they will all find nothing - but does that mean that there isn't any malware on the system?

This post is the first of several that will write, in an attempt to fully address this issue.

Characteristics
Before we start digging into the guts of detecting malware during dead box analysis, it is important to understand the four characteristics of malware, specifically the initial infection vector, the propagation mechanism, the persistence mechanism, and artifacts of the malware.  I originally developed these characteristics as a way of helping new analysts develop a confident, professional demeanor when engaging with customers; rather than reacting like a deer in the headlights, my hope was to help these new analysts understand malware itself to the point where they could respond to a customer request in a professional and confident manner.  Understanding and applying these characteristics enables analysts to understand, detect, locate, and mitigate malware within an infrastructure.

Initial Infection Vector
The initial infection vector (or IIV) refers to how the malware originally made it's way on to the system.  Worms like SQL Slammer took advantage of poor configuration of systems; other malware gets on to systems as a result of exploiting vulnerabilities in browsers.

Understanding malware IIV mechanisms not only provides us with a starting point for beginning our investigation, but also allows analysts to go from "I think it got on the system..." or "..if I had written the malware...", to actually being able to demonstrate, through empirical data, how the malware ended up on the system.  Too many times, the IIV is assumed, and that assumption is passed on the customer, who uses that information to make critical business decisions, possibly even validating (or, invalidating) compliance.

Also, keep in mind that many of the more recent malware samples appear on systems as a result of a multi-stage delivery mechanism.  For example, a user may open an email attachment, and the document will contain embedded malicious code that will exploit a vulnerability in the target application, which may reach to a server with additional instructions, and then the second stage will reach out to another server and download the actual malware.  As such, the malware does not simply appear on the system, and the IIV is actually much more complex than one might expect.

Determining the IIV can be an important factor in a number of exams.  For example, PCI exams require that the analyst determine the window of compromise, which is essentially the time from when the system was first compromised or infected, to when the incident was discovered and taken offline.  While this is done for fairly obvious reasons, other non-PCI cases ultimately have similar requirements.  Being able to accurately determine when the system was first infected or compromised can be a critical part of an exam, and as such should not be left to speculation and guesswork, particularly when this can be determined through the use well-though-out processes.

Propagation Mechanism
This characteristic refers to how the malware moves between systems, if it does.  Some malware doesn't move between systems on it's own...instead, it infects one system and doesn't move on to other systems.  For example, RAM scrapers found during PCI cases don't infect one system and then propagate to another...rather, the malware is usually placed on specific systems by an intruder who has unrestricted access to the entire infrastructure.

Some malware will specifically infect removable devices as a means of propagation.  Worms are known primarily for their ability to propagate via the network.  Other malware is known to infect network shares, in the hopes that by infecting files on network shares, the malware will spread through the infrastructure as users access the infected files.

It's important to note that the malware propagation mechanism may be the same as the IIV, but analysts should not assume that this is the case.  Some malware may get onto a system within an infrastructure as a result of a spear-phishing campaign, and once on the internal infrastructure, propagate via network or removable drives.

Artifacts
According to Jesse Kornblum's Rootkit Paradox, rootkits want to remain hidden, and they want to run.  The paradox exists in the fact that by running, there will be ways to detect rootkits, even though they want to remain hidden.  The same is true with malware, in general, although malware authors are not as interested in remaining hidden as rootkit authors.  However, the fact is that as malware interacts with it's environment, it will leave artifacts.


Looking for malware artifacts is a lot like using those "expert eyes" that Chris Pogue talks about in his Sniper Forensics presentations.  When malware executes, it interacts with it's environment, and the great thing about the Windows environment is that it records a lot of stuff.  In that way, it's a lot like using 'expert eyes' to look for deer on the Manassas Battlefield park...as deer move through the park, they leave signs or 'artifacts' of their presence.  They're grass eaters, and they leave scat that is different from fox, dogs, and bears (as well as from other grass eaters, like our horses).  They leave tracks, whether it's in the mud, sand, soft dirt or snow.  When they move through tall grass, they leave trails that are easily visible from horseback.  When they lay down for the night, they leave matted grass.  Sometimes the bucks will leave velvet from their horns, or if a couple of bucks are sparing, you may actually find a horn or two (we have a couple hanging in the barn).  My point is that I don't have to actually see a deer standing in a field or along a stream to know that there are deer in the area, as I can see their "artifacts".  The same thing is true for a lot of the malware out there...we just need to know what artifacts to look for, and how to look for them.  This is how we're able to detect malware during dead box analysis, particularly when that malware is not detected by AV scanners.

Some of the more popular locations where I've found artifacts include AV logs, and on Windows XP systems in particular, within the Windows firewall configuration.  I've seen a number of instances where malware has been detected by AV, but according to the logs, the AV was configured to take no action.  As such, as strange as it may seem, the malware infection is clearly visible in the AV logs.  In another instance, I was reviewing the on-demand scan AV logs and found an instance of malware being detected.  A closer examination of the system indicated that malware had originally been created on the system less than two days prior to the log entries, and I was able to verify that the malware had indeed been quarantined and then deleted.  About six weeks later, a new variant of the malware, with the same name, was deposited on the system, and throughout the course of the timeline, none of the subsequent scans detected this new variant.  Previously-detected malware can provide valuable clues of an intruder attempting to get their malware on to a system.

In other instances, the malware infection process creates a rule in the Windows firewall configuration to allow itself out onto the network.  Finding a reference to a suspicious executable in the firewall configuration can point directly to the malware.

In another instance, the "malware" we found consisted of three components...a Windows service, the "compiled" Perl script, and a utility that would shuttle the collected track data off of the system to a waiting server.  The first time that the malware components were installed on the system, within two days, an on-demand AV scan detected one of the components and removed it from the system (as indicated by the AV logs and subsequent lack of Perl2Exe temp folders).  Several weeks later, the intruder installed a new set of components, which ran happily...the installed AV no longer detected the first component.  This can happen a lot, and even when the system has been acquired, AV scans do not detect the malware within the mounted image.  As such, we very often have to rely on other artifacts...unusual entries in autostart locations, indirect artifacts,

The use of artifacts to detect the presence of malware is similar to the use of rods and cones in the human eye to detect objects at night or in low light.  Essentially, the way our vision works, we do not see an object clearly at night by looking directly at it...rather, we have to look to the side of the object.  AV scanners tend to "look directly at" malware, and may not detect new variants (or in some cases, older variants)

Persistence Mechanism
The persistence mechanism refers to the means that malware utilizes to survive reboots.  Some persistence mechanisms may allow the malware to launch as soon as the system is booted, while others will start the malware until a user logs in, or until a user launches a specific application.  Persistence mechanisms can be found in Registry locations...during the 2012 SANS Forensic Summit, Beth (a Google engineer) described writing a web scraper to run across an AV vendor's web site, and around 50% of the Registry keys she found pointed to the ubiquitous Run key.  Persistence can also be achieved through a variety of other means...locations within the file system, DLL Search Order parsing, etc.  Determining the persistence mechanism can be a very valuable part of gaining intelligence from your examination, and detecting new persistence mechanisms (to be covered in a future post) can been equally important.


As you may already see, the persistence mechanism of malware is also an artifact of that malware; persistence is a subset of the artifact set.  In many ways, this can help us to a great extent when it comes to detecting malware, particular malware missed by AV scanners.

It can be extremely beneficial to understand various artifacts that malware creates on a system, particularly in light of the fact that AV isn't catching everything.  Most AV appears to look for and then scan across executable files...some AV does find indicators based on text-based data, such as JavaScript code, etc.

Not all malware uses the Registry for persistence.  For example, Theola uses a Chrome plugin to perform bank fraud, and W32/Crimea modifies imm32.dll in order to remain persistent (I found this variant in 2010; this is a write-up from another variant from 2007).

Not all malware creates really obvious indicators in the Registry, either.  Corey talked about indicators for a variant of ZeroAccess.  However, I analyzed a system that had been infected with another variant of ZA, one that created the HKCU\Software\Microsoft\Windows\CurrentVersion\Ext\Settings\{8AD9C840-044E-11D1-B3E9-00805F499D93} key, the effect of which is explained in this GreyHatHacker blog post.  While this isn't a persistence mechanism, it does illustrate an indicator of a malware infection.

"Detecting" Persistence Mechanisms
There was a SANS webcast in January 2012 titled Detecting Persistence Mechanisms, during which a number of persistence mechanisms were mentioned, including several found in the Registry.  However, something that wasn't mentioned was how to actually go about detecting persistence mechanisms being created by malware.  Corey recently published an excellent blog post titled, Tracking Down Persistence Mechanisms, which does a great job of illustrating how easy it is to quickly examine the contents of autostart (or "ASEP") locations, particularly in the Registry.

The process I use to detect the use of Registry persistence mechanisms and other malware artifacts is to start by adding the key LastWrite times from the Registry hives (both NTUSER.DAT and USRCLASS.DAT for users) to my timeline.  This is exactly how I found the ZeroAccess artifact I described earlier in this blog post...the modified key was right there in the timeline.  I even went so far as to examine the hive file extracted from a VSC created just prior to the LastWrite time of the key, and I was able to determine that the LastWrite time was, in fact, when the key was created (i.e., the key didn't exist in the hive file from the VSC).  Timelines are a fantastic way to add context to the data that you're looking at, as well as to increase your relative level of confidence in the validity of that data.  However, timeline analysis is best utilized as part of an overall analysis plan, developing pivot points and items of interest via other data retrieval and analysis mechanisms.

Once I find an interesting Registry artifact in close proximity to other activity on the system that is associated with the malware, I have a number of options available to me.  Many times, I will open the hive in a viewer and take a look at what information is contained in the key itself...examine the subkeys, values and data.  I can correlate this with information gleaned from online searches, and very often, quickly write a new or modify an existing RegRipper plugin.  I then ensure that the Registry artifact is included as part of my shortened view of the overall timeline, along with a clear description of why it's significant, along with supporting documentation and references.  This serves the purpose of not only providing information to my customer, but also documenting the information for my own use.  In most cases, this entire process covers a span of a couple of minutes, maybe up to an hour depending up how much information is out there and available.

When Does It Start, and Why Does It Matter?
Where within the system that malware creates it's persistence mechanism has significant impact on your investigation, in part because investigations no longer center around the question of "was the system infected?" 

Take a look at this ThreatExpert report; the report points to a Registry value within the user hive that the malware adds data to in order to remain persistent, and then states:

...so that %AppData%\skype.dat runs every time Windows starts

IMHO, this can be easily misinterpreted.  If the analyst assumes that "Windows" refers to the system, then the statement is incorrect.  However, if "Windows" refers to the shell, then it is correct...but to a point.  In this case, the malware will start the next time that user logs into the system, and the Windows Explorer shell starts for that user. 

Ok...but so what?  Well, this can be a very important distinction to make.  For example, let's say that someone from the helpdesk logs into an account on a user's workstation in order to assist with or fix something.  They go to a web site to download a patch or update, and while it's installing, they do a bit of surfing...and the system gets infected.  If the malware infects the system within the context of only that user account (i.e., creates a Registry persistence mechanism in the "HKCU" hive), then that malware will not be launched again until that user account is used to log into that system again.  Where this distinction is important is in cases of the "Trojan Defense" (was the system infected, and did the malware execute?), as well as PCI forensic audits, where the PCI Council requires the analyst to identify the "window of compromise" in a dashboard area of the report.  For merchants that know about how many credit card transactions they have in a given time period, that "window of compromise" can have a significant effect on the overall outcome of the report, potential fines levied by the council, etc.  I examined a system once where the malware was identified and deleted by an on-demand AV scan less than 48 hrs after it was created on the system, and the intruder didn't upload a new version of the malware (albeit with the same name) for 6 weeks...which, like I said, had a significant impact on the overall outcome of the investigation.

In another example, I've seen a server systems that were infected with malware when an administrator logged in and performed some series of activities (usually web surfing or checking email...hey, it happens...) that led to the infection, with the persistence mechanism for the malware being in the Administrator user's NTUSER.DAT.  When the server is rebooted, the malware doesn't persist and begin running again until the user logs into that account...in some cases, depending upon the server, that could be for several days or weeks.  Once again, this is a very important distinction to make.

The ThreatExpert report mentioned above is only an example...it isn't the only site where these sorts of messages can be seen.  I've seen reports at the MMPC site that state that malware creates a persistence mechanism in the HKCU\..\Run key so that it "starts whenever the system starts".  The same is true for a number of reports at AV vendor sites.

Wow6432Node
I discussed Wow6432Node in a previous blog post, and Corey has discussed this as well. And yes, it is very important to point out yet again.  And again.  And again.  Why is that?  Because I honestly believe that most analysts are missing this source of data.