Threat Level: green Handler on Duty: Xavier Mertens

SANS ISC: InfoSec Handlers Diary Blog - Comparing Anti-Virus Solutions InfoSec Handlers Diary Blog


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!

Comparing Anti-Virus Solutions

Published: 2007-03-05
Last Updated: 2007-03-06 21:45:46 UTC
by Swa Frantzen (Version: 1)
0 comment(s)

Every so often we get requests from readers asking us about comparisons between the different anti-virus products. These requests range from recommendations on how to compare oneself over to ready made comparison reports.

Virustotal

Typically we tend to use virustotal output in a lot of the diaries we write as it gives a good overview where a given malware is detected and how the different vendors named it. E.g:

Antivirus Version Update Result
AntiVir 7.3.1.38 20070305 TR/Dldr.Small.ego.55
Authentium 4.93.8 20070305 -
Avast 4.7.936.0 20070305 -
AVG 7.5.0.447 20070305 Downloader.Generic3.VCI
BitDefender 7.2 20070305 Dropped:Trojan.Rootkit.AN
CAT-QuickHeal 9.00 20070305 -
ClamAV devel-20060426 20070305 -
DrWeb 4.33 20070305 -
eSafe 7.0.14.0 20070305 Win32.Small.ego
eTrust-Vet 30.6.3455 20070305 -
Ewido 4.0 20070305 Downloader.Small.ego
F-Prot 4.3.1.45 20070304 -
F-Secure 6.70.13030.0 20070305 Trojan-Downloader.Win32.Small.ego
FileAdvisor 1 20070306 -
Fortinet 2.85.0.0 20070305 W32/Small.EGO!tr.dldr
Ikarus T3.1.1.3 20070305 Trojan-Downloader.Win32.Small.ego
Kaspersky 4.0.2.24 20070305 Trojan-Downloader.Win32.Small.ego
McAfee 4976 20070305 -
Microsoft 1.2204 20070305 -
NOD32v2 2097 20070305 Win32/Wigon.K
Norman 5.80.02 20070305 W32/DLoader.CDZC
Panda 9.0.0.4 20070305 -
PandaBeta 9.4.3.3 20070305 -
Prevx1 V2 20070306 -
SAVMail 1.0 20070302 -
Sophos 4.15.0 20070305 Troj/Agent-ECZ
Sunbelt 2.2.907.0 20070305 -
Symantec 10 20070306 -
TheHacker 6.1.6.069 20070305 -
UNA 1.83 20070305 TrojanDownloader.Win32.Small.C329
VBA32 3.11.2 20070305 Trojan-Downloader.Win32.Small.ego
VirusBuster 4.3.19:9 20070305 -

File:
Name ccc.exe
Size 23040
md5 46241d432831fec22fd38c135ab96523
sha1 9d3dbf5c11779b4aceed2b2b2ff3735e9c483997
Date scanned 03/06/2007 00:52:27 (CET)

Obviously some vendors are absent from these results.

Virustotal keeps some limited statistics online, but they're not useful in comparing products.

Build your own

Now if you collect enough of these you might build your own statistics on which product detects things you encounter better than the competition. It's not easy to collect enough of them to get a statistically significant sample, so running 2 or more of your favorite scanners in-house might be easier to get more significant results -but more limited in scope-.

Getting enough malware to scan could be done using proxy logs, stripped email attachments etc. Do take care with local privacy rules/laws before doing this!

3rd Party Reports

There are some reports available about 3rd party testing of anti-virus products.

  • www.av-comparatives.org: updated every so often, includes a rating system.
  • www.pcworld.com: article, more than a year old.
  • www.av-test.org: runs out of a German University, not updated recently
  • www.virus.gr: last updated in August 2006
  • Consumer Reports has a comparison of 12 anti-virus products (subscribers only), it did get heavy negative feedback from the anti-virus vendors who seem not to like being put to the test.
  • Virus bulletin has a report online for registered users and is referenced by many of our readers.
  • Some more comparisons can be found at  antivirus-software.topchoicereviews.com and  www.consumersearch.com

Criteria

What's important to evaluate anti-virus products on? A test with a well known fake virus to see if it is detected (eicar), just will not expose the strengths and weaknesses of the different products and allow us to make a choice. Depending on the specific situation, we can be interested in:

  • Few false positives: detecting known good software as malicious and crippling systems as a result has happened before, the impact of recovering from this should not be underestimated. While looking forward is hard, the hindsight view might tell it's own tale
  • Few false negatives: not detecting malware is a bad thing, but it does happen by default in a technology that is basically reactive and where those creating the malware actually test their contraptions against the anti-virus products to make sure they are not detected at the time they release them.
  • Timely signature updates: signature updates is the main vector anti-virus software uses to fix the above problems. The faster they are released the more protection you get as a customer.
  • In corporate settings we want excellent centralized management. This should at least include a report that points us to individual machines not updating their definitions at all or in a timely manner. Ideally it does this without blocking signature updates when the roaming laptops are not connected to the corporate network or a VPN back to the office.
  • Few vulnerabilities: Vulnerabilities in our security solutions are somewhat of a nightmare as they not only fail in their goal of making us more secure but also introduce more security problems.
  • We do want variety if possible so that we use different engines in the different roles by e.g. using a different vendor on our email infrastructure and the desktops. The same goes for desktops and file servers.
  • Ease of use.
  • Price
  • ...

With thanks to epablo, Vincent,  Bryan, William, and many others for contributing to this diary

--
Swa Frantzen -- NET2S

 

Keywords:
0 comment(s)
Diary Archives