Tip of the Day: Surviving the monthly patch cycle

Published: 2006-08-09
Last Updated: 2006-08-09 22:31:42 UTC
by Swa Frantzen (Version: 1)
0 comment(s)
Yesterday we announced we would write about tips on how people patch their systems after a Black Tuesday. Since Mike is apparently suffering from a withdrawal symptoms after defcon, his fellow handlers will do the honors.

There are basically a few tactics to this in use. What strikes me in the responses we got: most of those writing in value not breaking applications significantly more than patching before you get hit with an exploit. Perhaps there is a lot work left to be done in order to convince (upper) management of the risks of patching late as patching even an hour after the worm or the targeted exploit hit you might cost the company significantly more than losing a few hours left and right over a not so critical system not being 100% healthy with a new patch.

Just patch

The folks doing this, take the risk and let the patches roll out in their organization. They expect a few systems to fail somehow more or less randomly and will deal with them as they go. Should one of the patches prove to be incompatible with one of their critical applications, they will deal with them at that point.

This group includes by necessity also most home users as they lack other means to patch. At best they can wait till others got some problems, but if all do that, it won't work and expose you for longer.

Test on limited scale, roll out carefully

You can use the Microsoft tools like WSUS and delay the rollout a few hours to make sure a few test systems survive and can run the critical applications. Typically smaller organizations use this with great success for the masses of general-purpose client machines.

Reader Ken wrote: "As we all know, patching any kind of operating system or application is fraught with dangers. In my environment, I don't have the luxury of a full test environment that I would love to have in order to be able to test each patch against all the applications and services in use. But that is just not possible with a limited budget.

In order to minimize the risk of a patch causing harm, I apply patches first to a set of known systems. The first system is my own workstation. I'd rather have it crash there than one of my coworkers' systems. After a day or so, the patches are then deployed to a subset of the systems (about 10) in the office. Finally, if there are still no issues found and no problems have been reported on sites such as the Internet Storm Center or on any of the security lists, the patches are distributed to all systems.
I actually use two tools for patch management. The first is Microsoft's WSUS service. I have all systems pointing there in order to get their updates. There are a couple of advantages. The first is Internet bandwidth usage. The patches are only downloaded once for all the systems. This can be a major savings in terms of time and bandwidth. Second, I can specify how and when the patches are applied via a GPO. Third, I control which patches are installed. If there is a known problem with a specific patch, I can just not release just that one patch to the users. Finally, I can get a status on which patches are applied and what systems have had problems installing the patches. The other tools is Shavlik's NetChk. This tool allows me to deploy a number of non-Microsoft Windows application patches and also to verify that patches are indeed being installed.

I use a similar process when it comes to apply patches to UNIX systems. First my own system, then a subset of system and finally all the systems.

So far, I have not had a major problem when it has come to applying patches. In almost every case where there was an issue, it surfaced within the smaller group of systems and the disruption was minimized.

Of course patching is not the only line of defense. I also have NIDs, firewalls, proxy web servers, virus scanning and log monitoring in place to try to reduce the risk to the office. Also recently, user information security awareness sessions have been started within the organization. This helps bring the users into the equation of defending the company against malicious software and web sites."

Test applications thoroughly

Testing applications to the end is next to impossible, you at the very best can test a few critical operations in your application and will have to gain trust it at some point. This approach is more often used on critical servers. The big drawback to this is that it takes time and resources to get this done. But in those cases where you end up with an incompatible patch you gain the pain of rolling back out incompatible patches and having to restore potential damage.

Mike wrote in on their strategy: "Simple strategy really:
  1. obtain patches, vet requirements and deploy to a QA environment, containing like for like hosts; exchange, SQL, IIS, workstation builds etc
  2. test, monitor, test, monitor...
  3. deploy to a pre-production group
  4. monitor, monitor, monitor
  5. deploy to primary production group
  6. monitor
  7. push out to remaining hosts/workstations.
The time scale for deployment is fairly aggressive with immediate introduction for 'critical' updates."

Personally I like the last line of his comment as it show they are trying to balance the heavy testing scheme with a fast track for getting those "PATCH NOW" patches out.

Fully features planned rollouts

Some organizations might (need to) plan ahead all their patching and actually do a roll-out plan that covers a long time before they come full circle and start over.

We had one such anonymous submission: "On the day after Black Tuesday, a task force meets to discuss the recently released patches. There is a set of ~100 users who represent all applications used. They get the patches via MS SMS to test. Once they verify their apps still function as expected, the patches are sent out via SMS each week to four predefined patch groups. This process lasts a month. Lather. Rinse. Repeat".

Divide and conquer

A well-known strategy from the real world can be used to divide the to be patched machines in different categories and tackle each differently. E.g.:
  • The general clients, not mission critical, could be patched as soon as the patches are available at Microsoft. This would yield some fall out left and right but just be ready to pick them up, those systems would get in trouble anyway. Why do we take the risks here? Well those systems might be your laptops that go the next day on a trip for 3 weeks and be used in the mean time in hotels, airports and other (potentially hostile) networks whithout a decent chance to get patched. Or they could be the laptop that takes off to a coffee serving place with annex hotspot and work there for a few hours, exposing themselves to any other visitor there. It gets worse if they pick up something evil and bring it home to a network of unpatched systems ...
  • The servers that are not mission critical, you could try to wait for the not so "PATCH NOW" patches, and roll them out if you see no problems reported, or you could just roll them out and be ready to roll back if you see problems. After all, it's not mission critical.
  • Mission critical servers should have many layers protecting them from evil, even from internal users. They should also not be exposed to most of the internal machines and they could remain unpatched or even isolated for a long while, till you get the chance to run the mission critical tests in a QA lab and roll out the patches being certain they don't break anything.
This approach does require a security architecture and approach that controls what computers are used for, and most environments are not ready for this today, but if you redesign your environment to a model that supports this, you can have it both ways at the same time.

Let's not forget that one of the reasons that getting Microsoft to release patches slow -aside from the obvious marketing impact- is that they test these patches. So you only get tested patches to start with ...

Thanks to all those writing in!

Swa Frantzen -- Section 66

Keywords: ToD
0 comment(s)

Detection and Deployment Guidance

Published: 2006-08-09
Last Updated: 2006-08-09 21:48:24 UTC
by Scott Fendley (Version: 3)
0 comment(s)
Microsoft released a Knowledge Base article yesterday that provides some guidance about deploying the patches from yesterday.  It does detail those updates that would not be detected, or deployed through the various mechanisms supported by Microsoft.   For more information, please see http://support.microsoft.com/kb/923752.

Update for those using Windows Update:

For those using the older Windows Update system, the Office patches are not supported under this system.  If you are using the newer Microsoft Update, the Office patches will be applied for the more recent versions (XP and 2003).  Office 2000 users will need to go to the Office Update site to get these patches or apply manually.

Update #2:
MS06-047 appears to only be partially installed by most all mechanisms supported by Microsoft.  The best bet appears to be to manually install this update on all computers.  Installing this patch through other mechanisms may leave a window of exposure in some cases.  We encourage you to read the above article for discussion on this patch.

Scott Fendley   ( sfendley -at- isc. sans. org)
University of Arkansas
0 comment(s)

A Peek Into The MailBag

Published: 2006-08-09
Last Updated: 2006-08-09 13:18:48 UTC
by Scott Fendley (Version: 1)
0 comment(s)
Good Morning Everyone!  We had a few items to just mention from the MailBag.

VA Desktop Stolen and VA Laptop Thiefs Arrested
As pretty well everyone knows by now, a desktop unit has been stolen which contained a number of records of vets.  This does not look good for the VA to have this second snafu so quickly after the previous one.  However, it is good news that law enforcement has arrested the teens involved in the laptop theft from  a couple of months ago. We usually do not report such  things as it has made it to CNN and is more of a political hot button at this point.  If you hear of something IT Security-wise from CNN, then chances are it is quite old news to the community, or has a political interest.  There is no need for us to copy the mass media. :-)

Websense Report on Phishing/Data Stolen via ICMP

Websense sent an alert a couple of days ago concerning the use of ICMP as a conduit for a new Phishing Trojan.  Is there any wonder that we recommend that you block everything inbound and outbound except those services you actually need.  Does every computer on your network need the ability to ping (or many other ICMP style activities) resources outside your network?  Probably not.   Be aware that the adversaries are finding those protocols we are not watching as closely and tunneling data in and out.   For more information on this, please see

Scott Fendley   ( sfendley -at- isc. sans. org)
University of Arkansas
0 comment(s)

Other Microsoft Updates Released

Published: 2006-08-09
Last Updated: 2006-08-09 11:56:44 UTC
by Scott Fendley (Version: 2)
0 comment(s)
Beyond the 12 Security Bulletins released today,  Microsoft released a few other updates that should be noted.

Update for InfoPath 2003 - KB920103
This high priority (non-security) update addresses some issues discussed in KB917510 and KB920914.  To the best we can tell, this is primarily a post Office 2003 SP2 reliability patch for the InfoPath product.

Malicious Software Removal Tool (MSRT) - KB890830
The MSRT underwent its monthly update to add detection for W32/Banker and W32/Jeefo.

Outlook 2003 Junk E-Mail Filter Update - KB920907
This update provides the Outlook 2003 client a more current definition of which e-mail messages are considered junk e-mail.

MS05-004 ASP.NET Path Validation Vulnerability Re-Release - KB887219
Those users of Microsoft Windows Server 2003 for Itanium-based systems or Windows Server 2003 x64 Edition should pay attention to this re-release bulletin.  Microsoft .Net Framework 1.1 Service Pack 1 is at rick for the Information Disclosure and possibly escalation of privileges these operating system environments as well.  The ISC recommends that this important update be applied as well.  (Thanks Stuart for bringing this re-release to our attention.)

Scott Fendley   ( sfendley -at- isc. sans. org)
University of Arkansas
0 comment(s)

Microsoft exploits on Reboot Wednesday

Published: 2006-08-09
Last Updated: 2006-08-09 11:55:47 UTC
by Swa Frantzen (Version: 1)
0 comment(s)
Well it certainly didn't take long for some to start making available (those I've seen so far are not for free) exploits against the vulnerabilities described in MS06-040, MS06-042 and MS06-046, which where only released yesterday.

Those of you're still testing patches, you'd better hurry up and get some of these fixed before you get hit.

Just as a reminder:
  • Filtering ports 135-139 and 445 helps against MS06-040; as do private VLANs (preventing client-client communication in the switch). None of those will help your fileserver, so patching is critical.
    Since there are still unpatched vulnerabilities in this software, filtering still remains crucial.
  • If you cannot apply MS06-042: stop using MSIE now, use an alternate browser.
  • Switching away to a browser not doing ActiveX (almost any will do) should help protect you against MS06-046 attacks as well.
But the best solution is to patch and do the above, layered defences!

Swa Frantzen -- Section 66
0 comment(s)


Diary Archives