The shortcomings of anti-virus software - SANS Internet Storm Center

The shortcomings of anti-virus software
No, this isn't about lousy detection rate. I think we're pretty much resigned to that, irrespective of the latest fancy marketing terms the industry uses to sell us the same failed concept. This is about the forensic quality, or rather lack thereof, of anti-virus. Let's say your anti-virus (AV) happens to find a Spyware. Something like the spyware that I described in yesterday's ISC diary. What does it do with it? If your AV is anything like the products that I've seen in use, it will display a Halloween-like scary pop-up ("Danger! Virus!") and will delete or quarantine the threat. So far so good. This used to be cool back when all we wanted our anti-virus to do was to get rid of the threat. But these days are over. Increasingly now, anti-virus alerts us (maybe) to a persistent threat that has been on the system for days, weeks, heck, even months. And deleting or quarantining such a threat causes a serious problem: It modifies or eradicates evidence. Yes, we get an alert, but then we are like the CSI guys who get called to a murder scene that doesn't have a body. Sure we can spend hours trying to lift DNA off cigarette stubs, but things would be so much easier if the caller could tell us what exactly he has seen where, and where the body was? In other words: If anti-virus removes a registry key to unhook a DLL, why can't the AV log tell me (a) where this registry key was and (b) when it was created? You know, this would give a first indication on how far back we have to dig to determine what data was stolen. The same holds true for the actual threat files that get deleted or quarantined: A full MAC (modify/access/create) timestamp in the logs shouldn't be too much to ask for? Maybe garnished with an MD5 checksum for good measure, so that the analyst can tell right away if the exact same threat has been seen on another PC already? I don't think the AV companies have caught on to this yet - they seem to be deleting and quarantining threats with the same casual indifference like they did 20 years ago, stomping all over the crime scene, and wiping out or contaminating important forensic evidence in the process. If your enterprise-grade anti-virus software does any better in forensics than described above, please let us know via the contact page. If it has the same shortcomings, please let us know as well, but more importantly, please let your AV vendor know. Maybe, someone listens.	Daniel 385 Posts ISC Handler Nov 2nd 2012
Thread locked Subscribe	Nov 2nd 2012 9 years ago
I've been learning about reverse engineering malware for a little bit, and have noticed the same problem. Fortunately, Carbon Black seems to fill in the gap. http://www.carbonblack.com/case-study-malware-infection-2/ http://www.sysforensics.org/2012/06/look-at-carbon-black.html	Anonymous
Quote	Nov 2nd 2012 9 years ago
Can't agree more given that some a/v forensics is such an essential requirement these days, especially in costing the time to rectify the aftermath and damage limitations. What is also made all the more difficult, is there is often no definitive 'list' of what files/keys should and should NOT be on the system at any point in time - dates, signatures can be spoofed, with certificates and any registry keys only secured in software installation restore points - all have been vulnerable to compromise. Building an image and wiping the disk with the saved image every day would be extreme to say the least! Highly impracticable given the number of registry reads/writes, would be to keep an accurate image of a complete disk installation and a copy at the beginning of each day. Subsequent comparisons could be logged for differences - but the overhead and analysis would likely be unacceptable. Logging is one way forward, but how do you protect the logs if the OS API calls and/or disk file system has been compromised? ps. notice VUPEN have reportedly found a number of 0-day vulns in Windows 8. Things can only get tougher without better tools.	Anonymous
Quote	Nov 2nd 2012 9 years ago
I believe Sourcefire's new anit-malware solutions fills these gaps. I'm going off marketing stuff though, I've yet to get my hands on it.	Anonymous
Quote	Nov 2nd 2012 9 years ago
Fireeye box does exactly that and more. Granted it should built in the endpoint security solution.	Anonymous
Quote	Nov 2nd 2012 9 years ago
In my experience, prevention is where to put the focus. Since AV is no longer effective at prevention, we need to significantly improve our prevention capability. When AV finds something, I consider that a failure of prevention. It's not that I disagree with you, it's just that there is only so much time and money to spend and the bad guys are winning. I think you do make a good point to the extent that AV forensics can help you identify chinks in the preventive armor. The (serious) limitation of AV presently is that it now finds such a small percentage of malware, it is also going to show you a correspondingly low percentage of your armor chinks.	PhilBAR 24 Posts
Quote	Nov 2nd 2012 9 years ago
Looking at it from their point of view why should they include forensics features in their products? The number of end users that would find this feature useful is surely very low. I doubt it would even be 1%. And unless law enforcement is going to start investigating malware at the endpoint level for everyone who reports malware, there is even less reason for them to do so. I suppose they might also take the case that forensics evidence might help the bad guys make a better product to avoid detection and removal.	KBR 63 Posts
Quote	Nov 2nd 2012 9 years ago
FireEye does do wonders, but on the endpoint, the closest thing I've seen is vSentry from Bromium. It is still in it's infancy, but it looks very promising. http://www.bromium.com/product/introducing-vsentry.html	KBR 6 Posts
Quote	Nov 2nd 2012 9 years ago
@Phil In my experience, detection is where to put the focus. Since passwords which used to be considered sufficiently strong are no longer effective at prevention, we need to significantly improve our detection capability... ;)	KBR 15 Posts
Quote	Nov 2nd 2012 9 years ago
Why are we still thinking that AV is a battle we can win? We know the number of samples is beyond the capabilities of blacklisting AV. The bad guys test there malware against all the major versions before deploying and the average zero-day doesn't get detected for what ~3 months?? Isn't it about time we gave up on trying to find all the bad and started whitelisting all the good? Shouldnâ€™t we refuse any code that the developer doesnâ€™t digitally sign â€“ so we know where it came from? Limit the allowed application to trusted companies? Further from that switch to whitelisting AV â€“ nothing is allowed to run thatâ€™s not approved. Weâ€™re aiming to lock down most of our company workstation: Weâ€™ll have trusted fileshares: \\company\installs \\company\executables \\company\deployments User can only install/run things from those locations; the whitelisting AV will block anything else. Gatekeepers in each area will be able to upload software to the fileshare â€“ the fileshare will run blacklisting AV to make sure itâ€™s clear. You can download virus.exe or virus.dll as many times as you like â€“ if itâ€™s not approved it canâ€™t run. ATMs have been doing it for years!	KBR 1 Posts
Quote	Nov 5th 2012 9 years ago
Thank you for bringing up this issue. Our in-house forensics team performs full forensic analyses on all hosts in our organization that are suspected of being infected/compromised (as identified by IDS, AV or other detection systems). One major gripe we've always had is that endpoint AV is constantly making the lives of our forensic analysts difficult for no good reason. When AV quarantines or cleans a threat it would be absolutely trivial for it to capture the necessary metadata (e.g., MAC timestamps, regkey creation times, etc...) that would allow for a proper forensic timeline to be built. Instead due to laziness/sloppiness, the vendors simply don't bother. This metadata is crucial information that anyone who is delving more deeply into any incident will need. By altering evidence without recording even the most basic of metadata it can completely (and unnecessarily) stymie an investigation into an incident. Worse still, if the AV has a bad file sitting in quarantine, even if the summary AV alert failed to collect or display the necessary information about the event, you'd think you could at least just remove the file from quarantine and examine it with its metadata intact. No such luck, that would make too much sense! Instead when the file is restored to disk, the AV allows the file system to completely alter the MAC information, meaning there's no way to view the original values! Again, it would be absolutely trivial for AV products to collect the necessary metadata but they simply refuse to do so. Our organization has brought this issue up many, many, many, times with various AV vendors whenever we have their reps in house. They even acknowledge that they've heard many other forensic organizations make this (trivial) feature request, but our pleas always seemingly fall on deaf ears. It's incredibly frustrating. On another note, forget about the whitelisting solutions that many in this forum seem to be alluding too as an alternative to standard AV. Just give it a little more thought and you'll realize that that's just trading one intractable problem for another.	KBR 1 Posts
Quote	Nov 6th 2012 9 years ago

No, this isn't about lousy detection rate. I think we're pretty much resigned to that, irrespective of the latest fancy marketing terms the industry uses to sell us the same failed concept. This is about the forensic quality, or rather lack thereof, of anti-virus.

Let's say your anti-virus (AV) happens to find a Spyware. Something like the spyware that I described in yesterday's ISC diary. What does it do with it? If your AV is anything like the products that I've seen in use, it will display a Halloween-like scary pop-up ("Danger! Virus!") and will delete or quarantine the threat.

So far so good. This used to be cool back when all we wanted our anti-virus to do was to get rid of the threat. But these days are over. Increasingly now, anti-virus alerts us (maybe) to a persistent threat that has been on the system for days, weeks, heck, even months. And deleting or quarantining such a threat causes a serious problem: It modifies or eradicates evidence. Yes, we get an alert, but then we are like the CSI guys who get called to a murder scene that doesn't have a body. Sure we can spend hours trying to lift DNA off cigarette stubs, but things would be so much easier if the caller could tell us what exactly he has seen where, and where the body was?

In other words: If anti-virus removes a registry key to unhook a DLL, why can't the AV log tell me (a) where this registry key was and (b) when it was created? You know, this would give a first indication on how far back we have to dig to determine what data was stolen. The same holds true for the actual threat files that get deleted or quarantined: A full MAC (modify/access/create) timestamp in the logs shouldn't be too much to ask for? Maybe garnished with an MD5 checksum for good measure, so that the analyst can tell right away if the exact same threat has been seen on another PC already?

I don't think the AV companies have caught on to this yet - they seem to be deleting and quarantining threats with the same casual indifference like they did 20 years ago, stomping all over the crime scene, and wiping out or contaminating important forensic evidence in the process.

If your enterprise-grade anti-virus software does any better in forensics than described above, please let us know via the contact page. If it has the same shortcomings, please let us know as well, but more importantly, please let your AV vendor know. Maybe, someone listens.

Daniel

385 Posts
ISC Handler

Nov 2nd 2012

I've been learning about reverse engineering malware for a little bit, and have noticed the same problem. Fortunately, Carbon Black seems to fill in the gap.

http://www.carbonblack.com/case-study-malware-infection-2/

http://www.sysforensics.org/2012/06/look-at-carbon-black.html

Anonymous

Can't agree more given that some a/v forensics is such an essential requirement these days, especially in costing the time to rectify the aftermath and damage limitations.
What is also made all the more difficult, is there is often no definitive 'list' of what files/keys should and should NOT be on the system at any point in time - dates, signatures can be spoofed, with certificates and any registry keys only secured in software installation restore points - all have been vulnerable to compromise.
Building an image and wiping the disk with the saved image every day would be extreme to say the least!
Highly impracticable given the number of registry reads/writes, would be to keep an accurate image of a complete disk installation and a copy at the beginning of each day. Subsequent comparisons could be logged for differences - but the overhead and analysis would likely be unacceptable.

Logging is one way forward, but how do you protect the logs if the OS API calls and/or disk file system has been compromised?

ps. notice VUPEN have reportedly found a number of 0-day vulns in Windows 8. Things can only get tougher without better tools.

Anonymous

I believe Sourcefire's new anit-malware solutions fills these gaps. I'm going off marketing stuff though, I've yet to get my hands on it.

Anonymous

Fireeye box does exactly that and more. Granted it should built in the endpoint security solution.

Anonymous

In my experience, prevention is where to put the focus. Since AV is no longer effective at prevention, we need to significantly improve our prevention capability. When AV finds something, I consider that a failure of prevention. It's not that I disagree with you, it's just that there is only so much time and money to spend and the bad guys are winning. I think you do make a good point to the extent that AV forensics can help you identify chinks in the preventive armor. The (serious) limitation of AV presently is that it now finds such a small percentage of malware, it is also going to show you a correspondingly low percentage of your armor chinks.

PhilBAR

24 Posts

Looking at it from their point of view why should they include forensics features in their products? The number of end users that would find this feature useful is surely very low. I doubt it would even be 1%. And unless law enforcement is going to start investigating malware at the endpoint level for everyone who reports malware, there is even less reason for them to do so. I suppose they might also take the case that forensics evidence might help the bad guys make a better product to avoid detection and removal.

KBR

63 Posts

FireEye does do wonders, but on the endpoint, the closest thing I've seen is vSentry from Bromium. It is still in it's infancy, but it looks very promising.
http://www.bromium.com/product/introducing-vsentry.html

KBR
6 Posts

@Phil In my experience, detection is where to put the focus. Since passwords which used to be considered sufficiently strong are no longer effective at prevention, we need to significantly improve our detection capability... ;)

KBR
15 Posts

Why are we still thinking that AV is a battle we can win? We know the number of samples is beyond the capabilities of blacklisting AV. The bad guys test there malware against all the major versions before deploying and the average zero-day doesn't get detected for what ~3 months??
Isn't it about time we gave up on trying to find all the bad and started whitelisting all the good? Shouldnâ€™t we refuse any code that the developer doesnâ€™t digitally sign â€“ so we know where it came from? Limit the allowed application to trusted companies? Further from that switch to whitelisting AV â€“ nothing is allowed to run thatâ€™s not approved. Weâ€™re aiming to lock down most of our company workstation:
Weâ€™ll have trusted fileshares: \\company\installs \\company\executables \\company\deployments
User can only install/run things from those locations; the whitelisting AV will block anything else. Gatekeepers in each area will be able to upload software to the fileshare â€“ the fileshare will run blacklisting AV to make sure itâ€™s clear.
You can download virus.exe or virus.dll as many times as you like â€“ if itâ€™s not approved it canâ€™t run. ATMs have been doing it for years!

KBR
1 Posts

Thank you for bringing up this issue. Our in-house forensics team performs full forensic analyses on all hosts in our organization that are suspected of being infected/compromised (as identified by IDS, AV or other detection systems). One major gripe we've always had is that endpoint AV is constantly making the lives of our forensic analysts difficult for no good reason. When AV quarantines or cleans a threat it would be absolutely trivial for it to capture the necessary metadata (e.g., MAC timestamps, regkey creation times, etc...) that would allow for a proper forensic timeline to be built. Instead due to laziness/sloppiness, the vendors simply don't bother. This metadata is crucial information that anyone who is delving more deeply into any incident will need. By altering evidence without recording even the most basic of metadata it can completely (and unnecessarily) stymie an investigation into an incident. Worse still, if the AV has a bad file sitting in quarantine, even if the summary AV alert failed to collect or display the necessary information about the event, you'd think you could at least just remove the file from quarantine and examine it with its metadata intact. No such luck, that would make too much sense! Instead when the file is restored to disk, the AV allows the file system to completely alter the MAC information, meaning there's no way to view the original values! Again, it would be absolutely trivial for AV products to collect the necessary metadata but they simply refuse to do so. Our organization has brought this issue up many, many, many, times with various AV vendors whenever we have their reps in house. They even acknowledge that they've heard many other forensic organizations make this (trivial) feature request, but our pleas always seemingly fall on deaf ears. It's incredibly frustrating.

On another note, forget about the whitelisting solutions that many in this forum seem to be alluding too as an alternative to standard AV. Just give it a little more thought and you'll realize that that's just trading one intractable problem for another.

KBR
1 Posts