A couple of years ago, I heard someone...a really smart someone...talk about "temporal proximity". I know that sounds Star Trek-y, but the reference was to initiating response activities as close to an incident as possible.

Several of the industry reports indicate that the majority of incidents that the vendors have responded to have been the result of third-party notification to the victim. That alone indicates a number of things, the lack of temporal proximity (perhaps a better description would be "temporal dispersion") being one of them.

Why is this so important? Well, a lot of good...no, strike that...a lot of critical information can exist in memory, making it very volatile. Processes complete, network connections terminate. The longer you wait, the more this happens...and the system is more likely to be rebooted.

Some intruders get on systems, run tools, and them leave with data. When they do so, they may delete their toolkits. I've seen batch files that include the "del" command for just that purpose. Well, the more temporal dispersion you have from the incident, the less likely you are to recover the deleted files...in case you haven't heard, Windows (especially XP) has it's own built-in antiforensics measures.

Okay, you're probably wondering what I'm talking about...so I'll tell you. For Windows systems, even if you don't interact with the system, stuff still happens, particularly with XP. Just let an XP system sit for a couple of days, and you'll see. Restore Points are created every 24 hours, and if the disk space available for the RPs is getting short, others will be deleted. A limited defrag is run every three days. And this is just for an XP system that sits there with no network connectivity and no one interacting with it. Now, add to that things like Windows software and application updates...you know, the stuff that just kind of happens automatically with a network connected system. Even with minimal auditing enabled, stuff still gets logged to the Event Log...more so on Vista and Windows 7, simply because there are so many more logs.

Now, add to the mix that no one within your infrastructure is aware of an incident (intruder, malware, etc.), and systems remain up, functioning, operational and in use. I've been on engagements where we collected data from a system and then three days later collected the same data...and you'd swear that they were two different systems. Prefetch files had been deleted, deleted files had been overwritten by OS and application updates, applications and tools being run, etc.

In order to achieve temporal proximity, you need a couple of things. First, visibility...if you don't have visibility into your infrastructure, how will you know when something occurs? You can't really expect to know when something goes wrong or changes if you're not monitoring, right?

Second, you need a plan. What's you're IR plan? Acquire memory and disk, and then take the system offline? Or panic and not do anything at all until someone who has no idea what's going on makes a decision? I can't tell you the number of times I've responded and found out that the incident had been detected a month prior, and the infected/compromised system had been left up the entire time.

Me: "You know the intruder has been siphoning data off of this system for the past month, right?"

Them: "We didn't know what to do."

This happens more than you'd care to know, and not just to one vertical...not just PCI, but to many, many types of victims.

One final note...Marines in training learn what are referred to as "immediate actions". These are simple tasks that you use to clear a jammed weapon. They're simple when you're on the range, on a bright, sunny day after a good night's sleep. You can ask a range coach if you're doing it right. But we're trained on this over and over because you never need it in those conditions...when you're going to need that reaction to be programmed is during an assault, at 2:30am, after you've gone without sleep for two or more days and maybe haven't eaten in as long. And it's raining. And it's cold.

Are your IT assets critical to your business? If I were to back up a truck and take all of your computers...all desktops, laptops, servers, etc...how would that affect your business? It would disappear, wouldn't it? Well, if IT assets are so critical to your business, why not protect them? The bad guys aren't coming into your organization and walking out with boxes full of papers...they're coming into your network and stealing data that way. And they're successful because in many cases, they have greater visibility into your infrastructure than you do.

Lenny Zeltser has an excellent presentation on his web site that discusses how to respond to the unexpected. This presentation is well worth the time it takes to read it...and I mean that not only for C-suite executives, but also IT staff and responders, as well. Take a few minutes to read through the presentation...it speaks a lot of truth.

When discussing topics like this, in the past, I've thought it would be a good idea to split the presentation, discussing responding as (a) a consultant and (b) as a full-time employee (FTE) staff member. My reasoning was that as a consultant, you're walking into a completely new environment, and as a member of FTE staff, you've been working in that environment for weeks, months, or even years, and would tend to be very familiar with the environment. However, as a consultant/responder, my experience has been that most response staff isn't familiar with the incident response aspect of their...well...incident response. FTE response staff is largely ad-hoc, as well as under-trained, in response...so while they may be very familiar with the day-to-day management of systems, most times they are not familiar with how to respond to incidents in a manner that's consistent with senior management's goals. If there are any. After all, if they were, folks like me wouldn't need to be there, right?

So, the reason I mention this at all is that when an incident occurs, the very first decisions made and actions that occur have a profound effect on how the incident plays out. If (not when) an incident is detected, what happens? Many times, an admin pulls the system, wipes the drive, and then reloads the OS and data, and puts the system back into service. While this is the most direct route to recovery, it does absolutely nothing to determine the root cause, or prevent the incident from happening again in the future.

The general attitude seems to be that the needs of infosec in general, and IR activities in particular, run counter to the needs of the business. IR is something of a "new" concept to most folks, and very often, the primary business goal is to keep systems running and functionality available, whereas IR generally wants to take systems offline. In short, security breaks stuff.

Well, this can be true, IF you go about your security and IR blindly. However, if you look at incident response specifically, and infosec in general, as a business process, and incorporate it along with your other business processes (i.e., marketing, sales, collections, payroll, etc.), then you can not only maintain your usability and productivity, but you're going to save yourself a LOT of money and headaches. You're not only going to be in compliance (name your legislative or regulatory body/ies of choice with that one...) and avoid costly audits, re-audits and fines, but you're also likely going to save yourself when (notice the use of the word when, rather than if...) an incident happens.

I wanted to present a couple of scenarios based on culminations my own experience performing incident response in various environments for over 10 years. I think these scenarios are important, because like other famous events in history, they can show us what we've done right or wrong.

Scenario 1: An incident, a malware infection, is detected and the local IT staff reacts quickly and efficiently, determining that the malware was on 12 different systems in the infrastructure and eradicating each instance. During the course of response, someone found a string, and without any indication that it applied directly to the incident, Googled the string and added a relationship with a keystroke logger to the incident notes. A week later at a director's meeting, the IT director described the incident and applauded his staff's reactions. Legal counsel, also responsible for compliance, took issue with the incident description, due to the possibility of and the lack of information regarding data exfiltration. Due to the location and use of the "cleaned" systems within the infrastructure, regulatory and compliance issues are raised, due in part to the malware association with a keystroke logger, but questions cannot be answered, as the actual malware itself was never completely identified nor was a sample saved. Per legislative and regulatory requirements, the organization must now assume that any sensitive data that could have been exfiltrated was, in fact, compromised.

Scenario 2: An incident is detected involving several e-commerce servers. The local IT staff is not trained, nor has any practical knowledge of IR, and while their manager reports potential issues to his management, a couple of admins begin poking around on the servers, installing and running AV (nothing found), deleting some files, etc. Management decides to wait to see if the "problem" settles down. Two days later, one of the admins decides to connect a sniffer to the outbound portion of the network, and sees several files being moved off of the systems. Locating those files on the systems, the admin determines that the files contain PCI data; however, the servers themselves cannot be shut down. The admin reports this, but it takes 96 hrs to locate IR consultants, get them on-site, and have the IT staff familiarize the consultants with the environment. It takes longer due to the fact that the one IT admin who knows how the systems interact and where they're actually located in the data center is on vacation.

Scenario 3: A company that provided remote shell-based access for their employees was in the process of transitioning to two-factor authentication when a regular log review detected that particular user credentials were being used to log in from a different location. IT immediately shut down all remote access, and changed all admin-level passwords. Examination of logs indicated that the intruder had accessed the infrastructure with one set of credentials, used those to transition to another set, but maintained shell-based access. The second account was immediately disabled, but not deleted. While IR consultants were on their way on-site, the local IT staff identified systems the intruder had accessed. A definitive list of files known to contain 'sensitive data' (already compiled) was provided to the consultants, who determined through several means that there were no indications that those files had been accessed by the intruder. The company was able to report this with confidence to regulatory oversight bodies, and while a small fine was imposed, a much larger fine, as well as notification and disclosure costs, followed by other costs (i.e., cost to change credit/debit cards, pay for credit monitoring, civil suits, etc.) were avoided.

Remember, we're not talking about a small, single-owner storefront here...we're talking about companies that store and process data about you and me...PII, PHI, PCI/credit card data, etc. Massive amounts of data that someone wants because it means massive amounts of money to them.

So, in your next visit from an auditor, when they ask "Got IR?" what are you going to say?

I caught a very interesting post on Brian Krebs' site this morning...you'll find it here.

As an incident responder, the first thing that caught my eye was:

Since the incident, he has conducted numerous scans with a variety of anti-virus and anti-malware products – which he said turned up no sign of malicious software.

Ouch! When I read things like that, I hope that it's not all (nor the first thing) that was done, and that it's a gross, over-simplification of the summation of response activities. Most times, though, it isn't.

I've read Brian's stuff for years, and I think that he's done a great job of bringing some very technical issues into the public eye without subjecting them to the glitz and hoopla that you see in shows like CSI. For example, while Brian mentioned some specific malware that could have been involved, he also made a very clear statement at the beginning of a paragraph that it has not been confirmed that this or any other malware had been involved. I think that's very important when presenting these kinds of stories.

So, look at the situation...the IT firm had a dedicated system with extra protective measures that was used to perform online banking. Even with those measures in place (I did some research on biometric devices back in 2001, and they don't provide the level of protection one would think), a bank official "...said the bank told him that whoever initiated the bogus transaction did so from another Internet address in New Hampshire, and successfully answered two of his secret questions."

I think that Brian's story is a very good illustration of what many of us see in the response community.

Malware may have been associated with what happened, but no one knows for sure. Many of us have been on-site, working with victims, and AV scans can't find anything, but the victims were clearly (and we later determine it to be true) subject to some sort of malware infection. It's interesting how an AV scan won't find anything, but check a few Registry keys and you start to find all sorts of interesting things.

Many of the "protection measures" that folks have in place are easily circumvented, or worse, lead the victims themselves to not consider that as an avenue of infection or compromise, because of the fact that they do have that "protection".

Finally, if malware was involved in this situation, it's a great illustration of how attacks are becoming smarter...for example, rather than logging keystrokes, as pointed out in the article, the malware will read the contents of the form fields; when it comes to online banking and some of the protective measures that have been put in place, this approach makes sense.

I've mentioned incident preparation before, and recently blogged on best practices, which was sort of a result of one of the comments to my first Incident Prep blog post. Specifically, the comment stated:

After having read the title I was really looking forward to this post, hoping I could compare our "CSIRP" against one you suggest - or at least making sure we've covered all the "must dos" and avoided the common traps...

So, this is that follow-up post...or one of them...that Anonymous asked about.

First, a couple of things...I can't suggest an online CSIRP example, per se, because there aren't many posted out there. However, I know absolutely NOTHING about Anonymous's infrastructure, so how can I recommend anything? Really? Ask me to recommend a good lunch, I know a wonderful carnivore-heaven/BBQ place nearby...but you may get there and say, "hey, I'm vegetarian!" Dude...you never said anything.

To address the other questions...again, knowing NOTHING about Anonymous's infrastructure or organization...the "must dos" are to have a CSIRP, and the common traps are to not have a CSIRP.

If you have a CSIRP and need a gap analysis, or if you need a CSIRP, here are a couple of things I can suggest to get you started...but again, these are all really dependent upon your organization, it's internal structure, as well as geographic, political and cultural considerations:

1. Is your organization subject to any legislative or regulatory oversight? Think HIPAA, PCI, etc. If so, start there. The PCI DSS v1.2, paragraph 12.9, has some information that specifies what a CSIRP should encompass, and is a good place to start...actually, for just about anyone, not just organizations subject to PCI. But the standard caveat applies...if it's associated with the PCI DSS, it's only a start.

2. CERT.ORG has an entire section on CSIRT Development.

3. Look at what you already have in place. Do you have a disaster recovery/business continuity plan? How about response plans for medical emergencies or in the event of workplace violence?

4. Keep in mind that a CSIRP is a communications plan, outlining who does what and when, in the event of an incident. Consider this...during CPR training, trainees are told to point to someone and say, "Call 911!" If there's a crowd gathered 'round and you just say, "would someone call 911?", what're the chances someone will do so? It's a good bet that everyone in the crowd will be thinking that someone else will do it...so you need to designate response staff. Think Star Trek, only without the dudes in red shirts.

Finally, and perhaps most importantly, if you're considering hiring outside consultants to assist with CSIRP development, you have to understand that this is NOT a drive-by service. What this means is that you cannot expect to hire someone to come in and drop off a CSIRP for you...at least, not without spending 6-9 months as a member of your staff. Why is that, you ask? Consider what I've said in previous posts...what is considered best practice for one organization may be completely undoable to another. It takes work, but ultimately your organization will have to take ownership of the CSIRP...after all, it's yours; your data, your infrastructure, your CSIRP.

In the course of my work, I will often encounter a customer's computer security incident response plan, or CSIRP...often, not always. In some cases, it may be that the customer had a CSIRP, and simply wanted validation of the plan and their processes, or a gap analysis. However, in most cases, responders such as myself encounter a complete lack of a CSIRP all together, which is an indicator that the organization we're assisting was not prepared for an incident at all.

In the end, the true impact of this fact is on the customer themselves (which, in may cases, may be passed on to their customers), who may have already been subject to an intrusion/compromise, and may be facing notification costs, fines...and maybe more.

The Art of Preparation
No, preparation for incident really isn't an art...I just wanted to get you to read further. The fact is, we all know what incident preparation is and consists of, because we do it all the time. An easy example of incident preparation is when we notice that the fuel gauge in our car is nearing the big "E". We anticipate a potential incident (i.e., running out of gas) was we take steps to prepare and mitigate the risks associated with an empty gas tank...we go to the gas station and fill up.

How about those of us who live in states where it may snow? We anticipate the risks associated with a driveway covered in snow, and we prepare to mitigate those risks; we get shovels, maybe some driveway de-icer compound, make sure we have a scraper available, etc. These aren't all the steps we may take...it really depends on where we live, and how willing we are to prepare.

So let's say someone lives in Maine or Minnesota, and makes a habit of NOT having a shovel, de-icer, tire chains, a full tank of gas, etc? Is this person prepared? Given that they're in a state with a high probability of it snowing, wouldn't it be prudent to take the necessary steps to prepare for an incident that has a high likelihood of occurring?

I would suggest to you that if your organization uses any sort of computing resources, the likelihood of you having (or, having had) a computer security incident of some kind is akin to that of it snowing in Maine in the winter months (and I know that this year, it already has!)...that is, the probability is rapidly approaching certainty. So why not be prepared?

Temporal Proximity
This is a term I heard a friend of mine use several years ago, and because I like stuff like Star Trek and the SciFi Channel, I keep it in the back of my brain housing group, ready to bring forth and assault my readers with. Oddly enough, it has a purpose here...that is, the closer proximity to the incident (with respect to time) that you begin collecting information and containing that incident, the greater your ability to really understand what's going on and address the issues of the incident. I'll use an example to illustrate what I mean...actually, a combination of several examples: a "victim" organization is notified of a breach of data by an outside third party, fully three months after the breach occurred. After about a week of trying to understand what could have happened, a responder such as myself is called in to assist. At that point, logs have rolled over and not been saved, systems have been taken out of service and reprovisioned, and IT staff is so busy that they can't remember what they had for breakfast, let alone what happened almost four months previously.

Another good example (by good, I really mean "seen often", not that the issue itself is good) is adding to the temporal dispersion by having relatively untrained staff conduct an "investigation" into the incident. By this point, systems have been scanned and rebooted (sometimes several times), patches installed, and again, some systems may have been rotated into or out of service. At this point, there is so much time (temporal dispersion) and activity between when the incident occurred and when any really meaningful steps are taken to respond to the incident, that the actual response activities are close to futile.

Consider an episode of your favorite variation of CSI, and let's say a crime occurred in a residence; if there is nothing mentioned about the crime for three years, and in that time, the residence has been burned to the ground, the structure completely razed and carted off to the dump, and a commercial structure built up in it's place, how is Grissom or Mac Taylor gonna to solve the crime?

Key Elements
Some of the key elements of Incident Preparation are your CSIRP, an understanding of your infrastructure (in particular, where your critical assets/data are located), and instrumentation. Without instrumentation, you have no visibility into what's happening within your infrastructure. Guys in submarines don't troll around the ocean depths without some sort of ability and instrumentation to determine where they are and what's going on around them. Instrumentation gives you visibility, and as such, decreases temporal proximity, particularly for intrusions or incidents of sensitive data leakage/theft.

Something in the news out of Tulsa, OK, this morning really provided an excellent lesson in IR.

Basically, the story goes that someone saw what they thought might be one of the deadliest spiders on the planet, panicked, and killed it. An expert in spiders asked to see the body of the spider, but it wasn't available...it had been destroyed.

How many times has this happened to you as a responder?

Caller: "Help! We were hit with the deadliest Windows worm known to man!"

You: "Okay, calm down. How do you know?"

Caller: "We received an alert on our AV console!"

You: "Okay, good. What did it say?"

Caller: "We don't know."

You: "Uhm...okay. Have you isolated any infected systems or preserved a sample of the malware?"

This is where things just kind of go downhill. But the news article is a great example of how things go wrong on a daily basis in IR...

When responding to an incident, the single biggest, most important factor is information.

Some of you are going to read this and your first thought will be, "well...duh!" Well, think about it for a moment...if you're a consultant (like me), how many times have you received a call for assistance where the answer to all of your questions was either "I don't know" or "blue"? How many times have you responded to an incident where the customer even had as much as a usable network diagram? Remember, like any (potentially) bleeding victim, they want answers NOW, but it's like trying to diagnose someone via email...whether you're on the phone or on-site, the first thing you need to do is orient yourself to the situation, which, of course, takes time.

In any incident, there are going to be unknowns; ie, a lack of information. At first, you may not know what you're dealing with, so some data collection and analysis (you know..."investigation") will be required, and this should be based on a solid process.

One of the unknowns during an incident should not be your environment; if it is, you're in trouble. By environment, I mean everything, include very basic stuff like "TCP connections start with a three-stage handshake." Laugh if you will, but I'm serious. Having JUST that basic piece of information and understanding how to apply it makes a tremendous difference during incident response, for anyone. Also, you need other information such as, where are systems located, and what are the paths that data is supposed to (or can) take? Where are applications located? Is there any logging, and if so, what is logged?

When it comes to responding to an incident, there are four main locations for data collection:
1. The network - classic packet sniffing; analyze with Wireshark, NetWitness, etc.
2. Network devices - routers, firewalls, any appliances (IPS, Damballa, etc.)
3. Host system/memory
4. Host systems/physical disk - includes not just the host OS, but any applications, application logs, etc.

Data can be collected from any of these sources to assist you in your incident, IF you know where they are, how to (or who can) access them, and where the necessary data is located.

During an incident, something that is more dangerous than a lack of information (because you can always fill the gap, often at the expense of time...) is misinformation. Sometimes the best thing a responder or incident manager can do is recognize that they don't know something, and then gather data and facts, and perform analysis, in order to fill the gap. The absolute worst thing that can happen is when those gaps are filled (or created) by speculation and blatantly incorrect information.

As an example, I've been involved in a number of malware-related incidents in which the corporate AV solution did not detect a bit of malware, or they were hit with a variant of known malware family. In one instance, the IT staff captured a copy of the malware and ran "strings" on it, and then searched Google for one of the strings they found. On the second page of the Google search, they found a reference to a keystroke logger on a public forum. Assuming that this was 100% credible, they proclaimed that the malware had keystroke logging functionality...which immediately caused the Legal and Compliance department to fire off a brown star cluster (Marines will find that one humorous) and declare an emergency! After all, what started out as a quickly-eradicated malware issue now became one of potential sensitive data theft!

In this case, the information collected by the IT staff (ie, the string) had no context. Where was the string located within the EXE file? In the resource section, or in the import table? Depending on which one, that string can affect incident response in vastly different ways.

In another example, I was assisting an incident manager and providing advisory services when I was watching a couple of the IT staff assembled in the "war room". Two of the IT staff were talking about something related to the incident, and I heard one of them mention "keystroke logger". Given the incident, having a keystroke logger on systems would be very bad...you might even say, super bad. Another IT staffer who was working away on the other side of the room looked up when he heard this, and said, "the Trojan is a keystroke logger?" Right about that time, an IT manager walked into room and heard this and made the statement, "The Trojan is a keystroke logger." An hour later at a status meeting, the IT manager reported to the incident manager that a keystroke logger had been installed on systems on the network. Hint: during the ensuing hour, no one had done any examination of either the Trojan or any of the systems.

During incident response, the key to effectively managing an incident is knowing what you don't know, and doing what's necessary to fill that gap. Hint: Speculating ain't it!

In my last post, we took a look at some ways to do malware detection, and in that post, I presented four general characteristics of malware that can be used to detect and deal with many of the issues that we run into. I thought that a good way to get discussion on this started would be to run through an example, the nice folks at the MMPC popped up a great example today...%LnkGet%. Let's take a look at the write up on %LnkGet% and see how we can apply the four characteristics, and see if a response methodology (and even an analysis methodology) begins to evolve...

Initial Infection Vector
Okay, the MMPC describes this bit of malware as a Trojan Downloader that when run, downloads other files. Also according to the MMPC, the initial infection vector is to arrive as an email attachment, so one way to begin looking for indications of this is to search for Windows shortcut (*.lnk) files in email attachment directories; remember, this may also include web-based email tools, as well, so looking for .lnk files in the web cache may be a good idea, too.

Artifacts
The artifacts of this malware are pretty straightforward...first off, the main file involved is a Windows shortcut/*.lnk file. Yes, there can be a great number of these on a Windows system, but as an analyst, what you'll be looking for is a .lnk file in an unusual place (ie, email attachment or web browser cache directory). In addition, this .lnk file downloads a .vbs script that then downloads additional malware.

From a network perspective, the downloading may leave artifacts in log files; as you can see from the write up, there are a number of sites involved which resolve to .cn and .tw domains.

Propogation Mechanism
This bit of malware is described as a downloader, and doesn't propogate on it's own. However, as a downloader, it may download additional malware that does propogate (ie, a worm of some kind).

Persistance Mechanism
This bit of malware doesn't seem to need a persistance mechanism, because once it's downloaded the additional malware, it's done. The MMPC does state that while the shortcut files do try to disguise themselves through the use of icons, they apparently do not delete themselves once they've completed their task(s).

What this tells us...
1. Block .lnk file attachments
2. Have a written policy and educate users against launching arbitrary files that they receive, regardless of the source.
3. Develop a CSIRP and train your responders (this is a subject for a whole other series of posts).

My reason for looking at things like this is that AV companies are not incident responders. People who discover vulnerabilities and publish exploits are not incident responders. In most cases, companies such as these provide information that they think is important, but very often what they provide is not sufficient for their users and customers to incorporate into their risk management and incident response planning. In this regard, I really think that these companies are doing their customers a huge disservice.