Chasing the DFIR Cure
I've wondered for some time now, how do "customers" (recipients of DFIR services) determine the quality and accuracy of the rendered product? Throughout my time in the industry, I've known some customers to "seek a second opinion", but is this something that's really pervasive?
I was watching a new medical mystery show recently called "Chasing the Cure". This show is all about folks with severe, debilitating illnesses that have gone un- or mis-diagnosed for extended periods of time. This subject is near and dear to me because about 25 years ago, I went through a similar situation, although the duration of my issue was a few months, rather than years. In one of the show segments, a woman was suffering from a debilitating ailment that had gone misdiagnosed for several years, and she was able to receive a confident diagnosis on the show (i.e., PCOS). What I found interesting was that the doctors who made the diagnosis (on the show) were looking at the results of tests that had been ordered by previous doctors, meaning that several medical professionals had the same data in front of them, and in the case, the woman had been told by previous doctors that did not have the condition for which she was finally diagnosed. So, this was not just a matter of two (or more) professionals having the same data and arriving at different conclusions, as much as it was about a professional in the field specifically discounting a diagnosis. The result was that the woman and her family suffered for several more years, where she could have received treatment much earlier. However, not being satisfied with the answer she was given led her to continue seeking a diagnosis.
That got me to thinking...how do those who contract for DFIR services know if the analysis and findings they're receiving are correct and accurate? Sure, if the findings don't go their way, they can seek a second (or third) opinion, but how do to they know that what they receive is correct?
Several years ago, I was contacted by an attorney who had a case that involved a person being in a specific location based on computer evidence. In short, the case had to do with someone claiming that they were at a convenience store, and this attorney's case would be held up if that person had been behind the computer keyboard. The attorney had first 'contracted' with a part-time IT sysadmin who serviced their office to conduct analysis of the computer data, and the sysadmin had reportedly found evidence that the person had, in fact, been behind the keyboard. The attorney asked me to confirm the finding, which I was able to do. However,
The question, "...are these findings correct?" ever asked? Does it matter? I believe it does, and I also believe that there are a number of circumstances where it may behoove a customer/recipient to seek a second opinion:
PCI Investigations - the "window of compromise" is a variable in the "potentially how many credit card numbers were compromised" equation. This then leads to corrective or punitive actions, such as fines, and as such, is significantly impacted by the findings from the investigation. For example, misinterpreting the time stamp associated with the AppCompatCache data can extend the "window of compromise" from weeks to years, and severely impact the merchant, who receives a much greater fine.
Compliance - this goes back to things like PCI investigations, as during the course of analysis, the analyst may find that the merchant was not compliant with the PCI DSS (or standards set by another regulatory body) at the time of the breach.
Cyber Insurance - the results of an investigation can significantly impact the results of a claim; issues with data collection and interpretation may lead to findings that indicate considerable gaps in "due diligence", and claims may not be paid.
HR - findings as a result of a DFIR investigation in support of HR can significantly impact the employee, or the company. Misinterpretations of the data may lead to an employee being unjustly accused or dismissed; I worked a pro bono case to this effect several years ago.
Ransomware - something we see reported quite often in the media is that "...there was no evidence of data exfiltration found...", and that's a good thing which fits the desired narrative. But is it correct? Were the DFIR analysts aware of the artifact locations within Windows systems that might provide a different view of that answer? After all, the actors behind Samas and Ryuk ransomware deployments have been observed spending months (yes, I did spell that correctly...) within an infrastructure before deploying the ransomware, so...yeah...
I'm not suggesting that the industry is rampant with errors in data collection and interpretation, not at all. There are a lot of great analysts out there doing a lot of great work, and providing accurate results and findings to their customers. However, like any industry, these things do happen, and like other situations, when they happen they can have a serious impact. We also have to look at the fact that operating systems are getting more sophisticated all the time, and applications are flourishing and getting more numerous. This is all to say that things are much more complex than they were 20 years ago, and with the number of people coming into the DFIR field, how do we keep up on keeping everyone at a common level of knowledge?
Another aspect of the industry that I've seen change over time is the use of collection, parsing, and pre-processing frameworks. When I started out in the industry, even if a DFIR analyst collected a dozen or more images, they analyzed those images themselves. Over time, as there's been a move to cover and address the enterprise, there's been a subsequent increase in the amount of available data. As such, in a move establish a level of consistency, a lot of DFIR teams have developed means for the enterprise-level collection and pre-processing of data. All of this can add an additional layer of abstraction between the data and the analyst.
I'm also fully aware that over time, we learn things. I was talking to Brett Shavers recently, and he brought up the scenario of going back and looking at previous cases. Like Brett, when I've done this, I've marveled at how far I've come since that case; what are some of the things I've learned since then that I could apply to the case if I were to address the issue today?
I would think, then, that without some compelling reason, most who purchase DFIR services accept the findings they receive as correct and accurate. In this age of legal and regulatory requirements that both impact and depend on the results of DFIR analysis, the correct and accurate collection and interpretation of digital data is paramount, and there are a number of cases where the "customer" may benefit from a second, or even a third opinion. After all, we do this with medical issues, don't we?
To that point, that 'compelling reason' would likely consist of findings that are markedly contrary and contradictory to the desired narrative. There is likely a 'threshold' that some may accept; for example, consider the PCI example above...there are likely merchants who receive information about those findings and are able to absorb whatever judgement is levied by the bank or the PCI Council. However, there are also those merchants for whom the judgement is what I've referred to as a "cratering fine"; that is, once the fine is levied, the business (a small mom-and-pop restaurant, for example) ceases to exist. I've seen this happen. In such cases, given what's at stake, it may behoove the merchant to seek a second opinion.
I was watching a new medical mystery show recently called "Chasing the Cure". This show is all about folks with severe, debilitating illnesses that have gone un- or mis-diagnosed for extended periods of time. This subject is near and dear to me because about 25 years ago, I went through a similar situation, although the duration of my issue was a few months, rather than years. In one of the show segments, a woman was suffering from a debilitating ailment that had gone misdiagnosed for several years, and she was able to receive a confident diagnosis on the show (i.e., PCOS). What I found interesting was that the doctors who made the diagnosis (on the show) were looking at the results of tests that had been ordered by previous doctors, meaning that several medical professionals had the same data in front of them, and in the case, the woman had been told by previous doctors that did not have the condition for which she was finally diagnosed. So, this was not just a matter of two (or more) professionals having the same data and arriving at different conclusions, as much as it was about a professional in the field specifically discounting a diagnosis. The result was that the woman and her family suffered for several more years, where she could have received treatment much earlier. However, not being satisfied with the answer she was given led her to continue seeking a diagnosis.
That got me to thinking...how do those who contract for DFIR services know if the analysis and findings they're receiving are correct and accurate? Sure, if the findings don't go their way, they can seek a second (or third) opinion, but how do to they know that what they receive is correct?
Several years ago, I was contacted by an attorney who had a case that involved a person being in a specific location based on computer evidence. In short, the case had to do with someone claiming that they were at a convenience store, and this attorney's case would be held up if that person had been behind the computer keyboard. The attorney had first 'contracted' with a part-time IT sysadmin who serviced their office to conduct analysis of the computer data, and the sysadmin had reportedly found evidence that the person had, in fact, been behind the keyboard. The attorney asked me to confirm the finding, which I was able to do. However,
The question, "...are these findings correct?" ever asked? Does it matter? I believe it does, and I also believe that there are a number of circumstances where it may behoove a customer/recipient to seek a second opinion:
PCI Investigations - the "window of compromise" is a variable in the "potentially how many credit card numbers were compromised" equation. This then leads to corrective or punitive actions, such as fines, and as such, is significantly impacted by the findings from the investigation. For example, misinterpreting the time stamp associated with the AppCompatCache data can extend the "window of compromise" from weeks to years, and severely impact the merchant, who receives a much greater fine.
Compliance - this goes back to things like PCI investigations, as during the course of analysis, the analyst may find that the merchant was not compliant with the PCI DSS (or standards set by another regulatory body) at the time of the breach.
Cyber Insurance - the results of an investigation can significantly impact the results of a claim; issues with data collection and interpretation may lead to findings that indicate considerable gaps in "due diligence", and claims may not be paid.
HR - findings as a result of a DFIR investigation in support of HR can significantly impact the employee, or the company. Misinterpretations of the data may lead to an employee being unjustly accused or dismissed; I worked a pro bono case to this effect several years ago.
Ransomware - something we see reported quite often in the media is that "...there was no evidence of data exfiltration found...", and that's a good thing which fits the desired narrative. But is it correct? Were the DFIR analysts aware of the artifact locations within Windows systems that might provide a different view of that answer? After all, the actors behind Samas and Ryuk ransomware deployments have been observed spending months (yes, I did spell that correctly...) within an infrastructure before deploying the ransomware, so...yeah...
I'm not suggesting that the industry is rampant with errors in data collection and interpretation, not at all. There are a lot of great analysts out there doing a lot of great work, and providing accurate results and findings to their customers. However, like any industry, these things do happen, and like other situations, when they happen they can have a serious impact. We also have to look at the fact that operating systems are getting more sophisticated all the time, and applications are flourishing and getting more numerous. This is all to say that things are much more complex than they were 20 years ago, and with the number of people coming into the DFIR field, how do we keep up on keeping everyone at a common level of knowledge?
Another aspect of the industry that I've seen change over time is the use of collection, parsing, and pre-processing frameworks. When I started out in the industry, even if a DFIR analyst collected a dozen or more images, they analyzed those images themselves. Over time, as there's been a move to cover and address the enterprise, there's been a subsequent increase in the amount of available data. As such, in a move establish a level of consistency, a lot of DFIR teams have developed means for the enterprise-level collection and pre-processing of data. All of this can add an additional layer of abstraction between the data and the analyst.
I'm also fully aware that over time, we learn things. I was talking to Brett Shavers recently, and he brought up the scenario of going back and looking at previous cases. Like Brett, when I've done this, I've marveled at how far I've come since that case; what are some of the things I've learned since then that I could apply to the case if I were to address the issue today?
I would think, then, that without some compelling reason, most who purchase DFIR services accept the findings they receive as correct and accurate. In this age of legal and regulatory requirements that both impact and depend on the results of DFIR analysis, the correct and accurate collection and interpretation of digital data is paramount, and there are a number of cases where the "customer" may benefit from a second, or even a third opinion. After all, we do this with medical issues, don't we?
To that point, that 'compelling reason' would likely consist of findings that are markedly contrary and contradictory to the desired narrative. There is likely a 'threshold' that some may accept; for example, consider the PCI example above...there are likely merchants who receive information about those findings and are able to absorb whatever judgement is levied by the bank or the PCI Council. However, there are also those merchants for whom the judgement is what I've referred to as a "cratering fine"; that is, once the fine is levied, the business (a small mom-and-pop restaurant, for example) ceases to exist. I've seen this happen. In such cases, given what's at stake, it may behoove the merchant to seek a second opinion.