Do we need test procedures in our companies before implementing Antivirus signatures?

Published: 2012-08-20
Last Updated: 2012-08-20 17:41:16 UTC
by Manuel Humberto Santander Pelaez (Version: 1)
13 comment(s)

We have heard a couple of cases regarding problems caused my faulty antivirus signature files.Most recend that has a worldwide impact was the Microsoft Antivirus treating code from google webpage as virus. In 2010, Mcafee deployed DAT 5958 which identified svchost.exe as a virus, deleting it an causing loose of network access. In April 2011, Mcafee deployed DAT 6329, which caused disruption in SAP telephone connectivity functionality as it recognized spsgui.exe with virus. Also deployed DAT 6682, which caused system crash in GroupShield Exchange (MSME), GroupShield Domino, and McAfee Email Gateway.

Yesterday, we received report from reader John stating that computers with DAT 6807 installed got conectivity problems. Today Mcafee confirmed this to be a problem if you are using VSE 8.8.x and have DAT 6807 or 6808 installed. Their recommendation is to go straight to DAT6809.

Other antivirus programs like AVG also deploys faulty updates. Since these events are becoming a worrying trend, should we implement test procedures inside our organizations as we do with other updates like the ones deployed by Microsoft with Windows Update? Implementing a faulty update has a high risk to the organization and its solution consumes considerable time and substantial resources. I am considering implementing such procedure for my company.

Do you think it's necessary to implement such procedure in your company? Let us know!

Manuel Humberto Santander Peláez
SANS Internet Storm Center - Handler
Twitter:@manuelsantander
Web:http://manuel.santander.name
e-mail:msantand at isc dot sans dot org

13 comment(s)

Comments

A similar problem for software developers is when antivirus software running at a customer sites quarantines an essential file and prevents the software from running. Antivirus makers typically do allow companies to submit malware-free reference files that should not be detected, but submitting these to all antivirus makers for all software and revisions can be tedious.
I don't know about other organizations, but we barely even have the time and resources for testing tons of OS and application patches before rolling them out. Who'd have time to do this for every rollout of a new DAT file? Anti-virus tools are already reactive in nature and (in my experience) new fingerprint rules are anywhere from a week to a month behind the appearance of new malware. Delaying this even further with continous testing would make anti-virus tools even less effective than they currently are (and they're a band-aid at best)
The cost-benefit analysis behind your eventual decision would be interesting. As your examples point out, bad signatures have had quite the range of effects. Pre-testing AntiVirus signatures isn't a straightforward mitigation though. If your staff spends more time validating software capability than they would have fixing an occasional glitch, this cure can be worse than the problem.

Delaying deploying the new signatures also adds some amount of risk - your systems remain vulnerable to the latest defendable malware for longer. After all that extra work, your internal testing may not even catch an incompatibility that shows up in production. I imagine that many organizations' approaches to automatic updates (of any sort) are ad-hoc. The "right" option depends on many technical and non-technical needs of each organization.
In my case, we have strong usage policies in place and our user environment is small, so having a delay in the distribution of the DAT files would be an informed and acceptable risk, within reason. Given that we are small and resources are very tight, I agree with Brent that spending additional time to start having to test AV signatures is not ideal. If the product is a paid subscription, I want the AV companies to be the ones to do more and better testing. Some of these things would have been found in internal testing as they were things that would affect any system. To not find such problems says they didn't effectively test the DAT before rolling it out. One would think that they would start to worry about their reputation as a reliable vendor when these incidents keep re-occurring.
McAfee EPO is our management tool for the VirusScan product from McAfee. We have setup a delayed deployment schedule where the latest DAT file release is downloaded in the evening to the Evaluation branch, deployed to a small group of test servers and workstations, then pulled down again the next morning into the Current branch then deployed over a staggered schedule trough out that day.

This process delays the DAT update roll out by 18 hours giving us time to validate the DAT update and for McAfee to pull a bad DAT back prior to it's release to our Enterprise. This has saved us during the bad DAT times as noted in the original article. If there is an issue, we are usually only one or two DAT releases behind but catch up during the day.
We stay one day back and it has saved us a few close-the-doors incidents over the years. But we consider desktop/server AV as the last resort anyway and we only have a 5% or 10% confidence factor in it. We use highly restrictive user rights, multiple layers of anti-malware at various chokepoints (firewall/IPS/SMTP gateway/email server/proxy server) so that something has to sneak past all of those first. That being said, we treat a desktop AV alert as a very high priority but there is well under one per week anyway. (2,000 nodes)
One size doesn't fit all here, but I'd say that an automatic but phased roll-out is a sensible trade-off. Watching Virustotal results "improve" over time for current threats, it looks like it often takes between 3-8 days for "protection" to arrive in the release pattern .. so I don't think tucking on 18hrs of delay with a phased roll-out is all that big a deal. Presumed of course that you have an establihed process to shortcut a rapid roll-out of the freshest beta pattern that you can get your hands on if things are really going haywire on your network. In other words .. this is a typical risk/benefit tradeoff that depens on the circumstances and situation.
My argument would be that it is the AV vendor's responsibility to test signatures before releasing them. That won't catch every incompatibility, of course, but false positives on files like svchost.exe seems to indicate that the vendor did absolutely no testing at all. McAfee makes some great products, but I view this as a chink in their armor.
Having a delay (a day) might be the best blanket option (of course everyone wants their own piece of the blanket). Yes a few specific programs or possibilities may not be figured out until you try it but let the rest of the world be your testers. If there is a problem, as McAfee has shown, they pull the bad DATs pretty quickly.

As most others have stated, doing a full, thorough round of testing for every single release would quickly find yourself getting increasingly behind real-time. At first you may be one behind the current release but eventually I think you'd find another newer current release would be out before you finished testing the first one and so on....
I have serveral VMs that represent the base OSs we have in use in our environemnt. On these gets installed every piece of software used in our environment. The boxes are setup to be patched just like anyhting else in the environemnt, and are also pushed new virus defs about 12 hours before they roll out to the rest of the company. These VMs are never used, but sit, logged into with a fake user account. The network to the VMs is locked down to only allow that traffic needed for patches and AV updates. If a bad patch OR AV definition is released, it will break one of these VMs 12 hours before it starts breaking production hosts.

Diary Archives