At a community college where I'm helping out whenever they panic on security issues, I recently was confronted with the odd reality of a lingering malware infection on their network, even though they had deployed a custom anti-virus (AV) pattern ("extra.dat") to eradicate the problem. Of course, these days, reliance on anti-virus is somewhat moot to begin with, our recent tally of fresh samples submitted to VirusTotal had AV lagging behind about 8 days or so. If you caught a keylogger spyware, 8 days is plenty to wreak havoc. I usually compare today's AV to the coroner in CSI, he can probably tell what killed you, but won't keep you alive.

But back to the college. Turns out they verify on a weekly basis if all the PCs have a current pattern, and they also verified that all their PCs got the "extra" pattern. The only problem was, their definition of "all" relied on the AV-tool itself. Obviously, if a PC doesn't have anti-virus installed, it won't show up on the anti-virus console. Hence, if your AV claims you have 100% compliance, you might want to check an alternate repository, like for example your Active Directory, to compare numbers. When I ran this test at the college, I found that their network/AD had 51 more workstations than their AV knew about. No wonder they still had frequent hits on the IDS for the backdoor traffic.

Never rely on a single security tool to tell you that everything is fine. Throw two or more sets of data against each other, and investigate discrepancies. Like your fishing or drinking or training buddy, security tools lie. Get acquainted with the usual pattern of lies (or obfuscated truths :), and surprises and disappointments will become less frequent.