Should We Still Test Patches?

Published: 2011-09-08
Last Updated: 2011-09-08 03:38:14 UTC
by Rob VandenBrink (Version: 1)
7 comment(s)

I know, I know, this title sounds like heresy.  The IT and Infosec villagers are charging up the hill right now, forks out and torches ablaze!  I think I can hear them - something about "test first, then apply in a timely manner"??  (Methinks they weren't born under a poet's star).  While I get their point, it's time to throw in the towel on this I think.

On every security assessment I do for a client who's doing their best to do things the "right way", I find at least a few, but sometimes a barnful of servers that have unpatched vulnerabilties (and often are compromised).

Really, look at the volume of patches we've got to deal with:

From Microsoft - once a month, but anywhere from 10-40 in one shot, every month!  Since the turnaround from patch release to exploit on most MS patches is measured in hours (and is often in negative days), what exactly is "timely"?

Browsers - Oh talk to me of browsers, do!  Chrome is releasing patches so quickly now that I can't make head or tails of the version (it was 13.0.782.220 today, yesterday is was .218, the update just snuck in there when I wasn't looking).  Firefox is debating removing their version number from help/about entirely - they're talking about just reporting "days since your last confession ... er ... update" instead (the version will still be in the about:support url - a nifty page to take a close look at once in a while).  IE reports a sentance-like version number similar to Chrome.

And this doesn't count email clients and severs, VOIP and IM apps, databases and all the stuff that keeps the wheels turning these days.

In short, dozens (or more) critical patches per week are in the hopper for the average IT department.  I don't know about you, but I don't have a team of testers ready to leap into action, and if I had to truly, fully test 12 patches in one week, I would most likely not have time to do any actual work, or probably get any sleep either.

Where it's not already in place, it's really time to turn auto-update on for almost everything, grab patches the minute they are out of the gate, and keep the impulse engines - er- patch "velocity" at maximum.  The big decision then is when to schedule reboots for the disruptive updates.  This assumes that we're talking about "reliable" products and companies - Microsoft, Apple, Oracle, the larger Linux distros, Apache and MySQL for example - people who *do* have a staff who is dedicated to testing and QA on patches  (I realize that "reliable" is a matter of opinion here ..).  I'm NOT recommending this for any independant / small team open source stuff, or products that'll give you a daily feed off subversion or whatever.   Or if you've got a dedicated VM that has your web app pentest kit, wireless drivers and 6 versions of Python for the 30 tools running all just so, any updates there could really make a mess.  But these are the exceptions rather than the rule in most datacenters.

Going to auto-pilot is almost the only option in most companies, management simply isn't paying anyone to test patches, they're paying folks to keep the projects rolling and the tapes running on time (or whatever other daily tasks "count" in your organization).  The more you can automate the better.

Mind you, testing large "roll up" patch sets and Service Packs is still recommended.  These updates are more likely to change operation of underlying OS components (remember the chaos when packet signing became the default in Windows?).

There are a few risks in the "turn auto-update on and stand back" approach:

  • A bad patch will absolutely sneak in once in a while, and something will break.  For this, in most cases, it's better to suck it up for that one day, and deal with one bad patch per year as opposed to being owned for 364 days.  (just my opinion mind you)
     
  • If your update source is compromised, you are really and truly toast - look at the (very recent) kernel.org compromise ( http://isc.sans.edu/diary.html?storyid=11497 ) for instance.  Now, I look at a situation like that, and I figure - "if they can compromise a trusted source like that, am I going to spot their hacked code by testing it?"  Probably not, they're likely better coders than I am. It's not a risk I should ignore, but there isn't much I can do about it, I try really hard to (ignore it


What do you think?  How are you dealing with the volume of patches we're faced with, and how's that workin' for ya?  Please, use our comment form and let us know what you're seeing!
 

===============
Rob VandenBrink
Metafore

Keywords: patches
7 comment(s)

Comments

I don't think it's my job to do software testing for every piece of software we've purchased. That's part of why we paid for the software. For simple patches I primarily rely on two kinds of testing: monitoring and anecdotal.

For servers, if it's important then it's being monitored, so the test is to simply roll out the patch and make sure the monitoring says it still works. Ideally this is done at a time when restoring backups or rolling back to snapshots would be least disruptive.

For client side stuff, we don't run completely locked down clients so there's really very little we can do. Easiest thing is to get the patches out to the users and wait for a phone call if something happens to go wrong. Historically, our users get malware much more often than they get bad patches. Since the patches stop the malware and the malware doesn't usually come with a nice uninstaller like a patch does, I'd rather deal with patch problems.

As for the possibility of a compromised update source, I tend to lag behind by a couple of days anyway. Since I watch sites like this one, I'm generally going to find out about anything like that before the update goes out (or nobody finds out and it wasn't something I could prevent anyway).

We are seeing many issues with patching/upgrading.
Acrobat X changed many things, and we have issues with supplier websites where we can't print out the lables they try to deliver in PDF.

Everything talking to government systems is dependent on you having an old vulnerable Java version. If we upgrade Java, we might not be able to do what we need.

So we have 2 products that has a large marketshare and large exploits list, that we can not patch. We don't know how long we will be stuck on old versions.
Your posting should help reinforce the idea that prevention is simply not enough. No matter how many patches you apply, or IPSs or anti-malware systems you install, you will still have vulnerable systems. Organizations need to put more effort into detection and response efforts.

If ... I mean when.. you get owned, how will you know, and what will you do about it?

If you can't answer those questions, you can't even begin to claim you're protecting the network
The best way I've heard to think about this comes from our CIO (I know, clueful management - it's a Good Thing[tm]). At heart all operations and especially security operations is risk management. So weigh the impact and probability of a compromise for a given patch set and plan accordingly (for example, use the ISC cheat sheet each month :)).

Our approach is to patch everything non-production as well as "small" production systems ASAP. Then test only the things we care about before we patch the "big" production stuff (which is only a day later). The idea is not so much we would skip a patch, but that by testing ahead of time we can figure out if there are workarounds we need in place after it is applied for those systems that are most visible/accessible/vulnerable/important.

And also concur with Ron - you still need to assume compromise will occur no matter what you do and engineer your networks/systems to be able to notice and to mitigate the impact when it occurs.
At the company I work for, the position that is taken is that patches *must* be tested before being released into production. Given the kind of work that we're involved in, and the fact that a stop on the production line of any extended period would significantly impact the bottom line and the delivery schedules, the decision is more to protect the infrastructure from outside attacks, and take the time to test and verify that the patches won't break the production software in use. It's never a good trade-off, as there are internal and external sites that still use older versions of plug-ins and software, not to mention dealing with government requirements for certain versions of applications and software... it can be and generally is a nightmare to manage. But in general the company does a good job in keeping the vulnerabilities to a minimum.
It depends on what you mean by "test". I manage locked down workstations (I don't even run with local admin on my machine - I insist on dog fooding).

Microsoft patches get packaged, install tested (do the patches install and does the scan engine report everything as patched) on VMS, and then advertised en-masse within 4 hours of release, with the install set to go mandatory at 4 PM on Wednesday. Yes, that gives us 36 hours of vulnerability. But it gives me a set of 10 to 20% of my user base who will install the patches when they first pop up as beta-testers. That gives me 24 hours to halt things if nasty problems crop up.

Non-Microsoft patches have to get packaged (remember most auto-update systems won't work without local Admin, and even then vendor auto-update doesn't give me tracking) for deployment. I run through some validation stuff to see what files and registry entries are changing, verify that the new install routine does the same thing as the old install routine+update routine, and verify that the update routine works whether or not the user is logged in and whether or not the user has the app/plugin in question under use. That takes anywhere from 2 to 6 hours (Adobe Reader/Acrobat is such a pain because installing Acrobat 9.4.5 takes over an hour to install the base install and then EVERY single patch, since Adobe doesn't do cumulatives). The next morning, I advertise to a limited beta-pool (5% of the machines) with the job set to go mandatory at 4 PM the same day. Beta is to test both that my packaging runs on production boxes successfully and to let the users look for issues. Then I let the machines soak for a day, then advertise the job to the masses the following morning and give em until 4 PM the following day to install. So if an update releases on Monday, beta install happens Tuesday, Wednesday is soak, advertise happens Thurs morn, and patch goes mandatory Fri eve. If I have intel that indicates the vulnerabilities in question are under attack, I'll eliminate the soak day and run the install with only one day before mandatory (i.e. advertise Wed morn, mandatory Wed eve). Or go even more aggressive if I'm really worried.

So my testing is pretty much limited to, "Does it install, does it modify the files I think it should, does the plugin function against one simple test page." I have a whole testing framework that handles taking snapshots of the files and registries on the test VMs, verifying pre-conditions on the VMs, creating one-off SMS advertisements, watching for the jobs to run and finish, doing the post-install snapshots, copying the snapshots off and diffing them, etc. It even has a rudimentary notification service to tell me when to logon to the machine, when to start a browser for plugin update testing, when to logoff, when the test vector has finished, etc. I still have to trigger the various vectors, but 95% of what I used to do by hand is automated. Given that I package and deploy something around 180 software updates a year, and I have other responsibilities as well, this is a must.

We use a pretty funky build process (only thing in the image is the OS and hardware drivers - everything else gets installed from the SMS servers, so I don't have to keep generating new hardware images constantly), and the result is a dependency driven build process without one defined sequence, so I try to test a small variety of build scenarios after the Patch Tuesday updates. The build process test vector takes around 14 hours to run, but if I didn't make any mistakes and there are no hiccups, all I have to do is start it and wait. Otherwise, it'll pause when it spots an error and wait for me to correct it (or restart it if needed).

I let the beta-testers (who generally don't even know that they are beta-tester) act as guinea pigs, and hope that if they have problems the call in before I flip the switch.

It's all about balancing risk in a bunch of directions, and given I'm only one person, th reality is that any real testing I tried to do would take way too much time and still probably not spot the problems. So I let a small cadre of users take the risk for the rest of the population. Still, my guestimate is that I'm probably in the top 10% of corporations in terms of how fast I turn around patches/plugin updates and get to en-masse coverage.
I actually wrote about patching back in 2008 http://www.informit.com/articles/article.aspx?p=1244449

With the quick reverse-engineering that attackers do now on patch releases, I want my patch deployed ASAP. So my goal was:

- patch release on "Black Tuesday"
- deploy to test lab Wednesday
- check for problems Thursday
- deploy to production on Friday
- Secure for the weekend after patch deployment

The problem is, many system owners are "gun shy" and want complete regression testing and weeks in the lab before they touch production. 30 days is common in many places, and today that's too much exposure to risk, IMHO.

Diary Archives