I recently had a routine "can you help our business partner" type call from a client. Their business partner could receive email from them, but could not send email to them.

After a bit of digging in the SMTP header of a failed note, it turned out that the business partner was running a very old version of QMAIL, which has a problem with ESMTP and DNS responses larger than 512 bytes. My client (the destination for the email) had recently gone to an email scanning service, so the total return on an MX record request was well over 1.5kb.

So far, not so exciting, you say - patch the server and be done with it! So why am I writing this up on isc.edu?

This is where it gets interesting. I called the business partner, and their verbatim response was "Gee, I don't know. Applying that patch will involve taking the mail server down, our CEO won't approve that. Is there some other way to do this?"

Wait, what? Did I hear that right? Let me check my watch - what century is this again? This is a patch from 2007 for goodness sake! I can see needing to follow a change control procedure, schedule an outage, maybe for after-hours, but they are an application development shop, not the Department of Defense! If they're running a mail system that hasn't been patched in 4 years, chances are that someone else already owns them, and they've got bigger problems than just this.

Anyway, after a frank and honest (and tactful, though that part was a bit more difficult) discussion, they did apply the needed patch, along with a truckload of other system updates that had been delayed since forever.

I've encountered a few situations where it makes some snse for system admins to defer patching for extended periods of time:

Servers that support military personnel in active operations are often mandated by policy as "frozen". In our current global environment, these freeze periods can extend into months and years.

Servers that support long-range space exploration missions will often end up running operating systems that are no longer supported, on hardware that has been end-of-lifed years ago, or on hardware or OS's that were one-shot custom efforts. In cases like this, the hardware is generally air-gapped or otherwise isolated from sources of attack.

Some servers in support-challenged situations might also be "frozen" for specified periods of time - if I remember correctly, the servers in some of the Antarctic missions (really, no pun!) are in this category. (If I'm mistaken on this example, I know that sysadmin for those systems is a reader, please correct me!)

So the question I have for our readers is: What situations or applications have you seen that might defer patches and updates for an extended periods of time? Did you consider those reasons or policies to be legitimate? Did you come up with a compromise or workaround to get patches applied, or did you have to follow policy and not apply updates? Did this end up with a system compromise, and if so, did the policy protect the system administrator, or did they end up taking the blame anyway?

I'm really looking forward to feedback from our readers on this, please use the contact form to let us know what you've seen!

===============

Rob VandenBrink Metafore