SANS ISC: Change is Good. Change is Bad. Change is Life. - SANS Internet Storm Center

• SANS Site Network
  • Current Site
  • Internet Storm Center
  • Other SANS Sites Help
  • Graduate Degree Programs
  • Security Training
  • Security Certification
  • Security Awareness Training
  • Penetration Testing
  • Industrial Control Systems
  • Cyber Defense Foundations
  • DFIR
  • Software Security
  • Government OnSite Training

SANS ISC InfoSec Forums

In a lot of ways, our job in IT and Information Security is implementing change.  But as we all know, every change involves risk, and changes gone bad can be your worst nightmare.  I’ve seen the number of business system service interruptions due to changes in infrastructure pegged at anywhere from 50 to 70%, and I’d lean towards the high side of that.  If it’s not a hardware failure, it’s probably a mistake.  Why is this number so high?'

Testing takes time, time that we often don't have.  Managers will often consider every change to be "simple", so getting testing time is often a challenge.  We'll often be working in complex environments, environments that involve lots of people who don't care about your change until it breaks something of theirs.  We often have system components in the mix that we don't know about until something breaks.  We may have legacy apps that simple cannot be known - stuff that was written 20 years ago by people who have long since retired for instance..  Compound this with the pervasiveness of the GUI interface , which in a lot of ways encourages people to stumble through the menu until they find a setting that works (yes this happens, every day).

So what to do?  It’s our job to patch and update systems, to take projects from a set of ideas to working systems.  All of this involves change, every day.  Most of us have some method of change control.  This generally involves forms and meetings, but don’t run for the hills, it’s not all bad – really most of it is common sense:

•    Figure out what you’re doing
•    Make sure it’s going to work
•    If it messes up, have a plan
•    Clear it with others in your team to make sure it won’t mess anyone else up (remember, they’re all making changes too)
•    Make the change
•    Tell your team mates that it’s done.

You can do all of this with a few emails, but as I mentioned, most of us have forms to make sure we don’t forget steps in this, and we have meetings to make sure that everyone who needs to know gets told (and doesn’t have the “but nobody told me” excuse later).

Just the facts
So let’s start with the form.  It’s simple really.  You need to describe your change in terms that your audience will understand.  Your audience will usually include folks are technical, but don’t always understand your job.  Sometimes non-technical people will be on the list – business unit people who may be affected for instance.  Whoever is on the list, be sure that you’re accurately describing the change, but aren’t talking down to them.

You’ll need a plan.  This needs to describe both the change and how you are going to test it.  I almost always use a Gantt or Pert chart to describe my plan.  Not only does everyone understand a picture (remember your audience?), but at 2am when I’m doing it for real, it’s REALLY easy to miss a step.  And an hour into it, what you DON’T want to do is undo steps 4 through 15 so that you can do the step 3 that you missed the first time.

You need a backout plan.  I don’t know about you, but lots of things that I do fail in spectacular ways – this is what makes my job so much fun.  The important thing is to have a plan to deal with it.
Put a time estimate on your plan, and be sure that the change window you are requesting includes the time to make the change, test it, then back out.  If you have a backout plan, then problems become what they should be - an opportunity to learn something new, rather than the political firestorm that they can become if you don't have that plan ready to go.

What is a change?  Especially when change control is new in an organization, the topic of what change is will be a hot one - when do you need to fill out a change control request?  I've been in organizations where unplugging an unused ethernet cable is a change, or plugging in a power cable.   On the face of it, you might think that these are both going overboard.  But if you unplug that dead cable you may find out later that it's not really dead - it's only used to run that legacy accounting report at month end.  Or when you plug in the new IPS unit you may trip the breaker on the PDU that the main firewall or fileserver is plugged in to.  The level of change that needs to hit the process will vary from company to company, if you're just starting out in this, no worries, just be prepared for some healthy back-and-forth as you find the level between "too much paper work" and "enough process to get the job done"

Meetings, Meetings and more meetings – OK, just one meeting, but BE THERE.
After you’ve got the change all mapped out, you’ll need to make sure that everyone who needs to know, knows.  The whole point of change control is to avoid conflicting changes.  Just to pick a few examples - If you’re doing a firewall upgrade, it can’t be when the DBA needs to VPN in to do a schema update.  Or if you’re doing a router upgrade, it can’t be during a server migration that’s on the other side of the router (yes, I’ve seen both at one company or other).
 

The meeting before the meeting
So you need a meeting, but not immediately.  In most cases, change control forms need to be submitted a few days before the meeting.  This is so everyone who needs to has a chance to review the change and ask any questions.  By the time you get to the meeting, you should have informal approval.

Finally – the meeting
The point of a change control meeting is to get formal approval and signoff.  You need to know that everyone is ok with the change, and that it won’t conflict with anyone elses change.   This means a few things – you need a quorum, and you need the right people there.  If your DBA boycotts your meetings, then there’s no point in having them.  Really everyone in IT needs to buy into change control, because if they don’t, it’s their changes that will bite your company in the tender parts!

Change control meetings should be SHORT.  I normally see discussion limited to 5 minutes per change – discuss any last minute details, be sure that change A and Change B don’t conflict, then it’s a “yes” or a “no”.  If you go over 5 minutes, either you didn’t do your homework (remember the meeting before the meeting?), or someone is hijacking your meeting.

Speaking of hijacking, change control meetings need a chair.  In many companies, the chair rotates between everyone who is NOT a manager.  This is a good thing for the team, as it gives people who otherwise wouldn’t speak during a meeting practice in talking to actual people.  It also emphasizes that the meeting is to make the job of the  "doers" easier - at its root change control is not supposed to be a heirarchal management function at all.  On the other hand, in lots of other companies the IT manager chairs the meetings.  In either case, the main job of the meeting chair is to keep it moving, to be sure that changes are approved or nixed quickly so everyone can get back to work.

Finally, any completed changes should be discussed.  The point of this is to continually make the process better.  Reviewing completed changes is all about making sure that the testing process is good, and that people are writing good implementation and backout plans.

Doing the Deed
When you are making your change, follow the plan.  Sneaking in little changes here and there will only cause you grief.  If you have an hour approved, the backout plan is 10 minutes and you are at 50 minutes and still typing, you should bail.  Going over on changes is an unscheduled outage, which everyone takes  a dim view of.  If you needed 70 minutes instead of 60, or even 2 hours, ask for that up front.  If you've got a plan and a process, this all get simple, and you get to make that change (aka - real work) in peace !

I wrote this diary because I end up talking to one client or other about change control processes a few times per month - it's a pretty popular topic (especially when you're convincing people who don't have a process that they need one).  I've attached a a sample change control form below.  If you find them useful,  feel free to modify this and use it in your organization (or not).   Over the years, I've seen change control implemented as Word documents in a shared folder or paper binder (this doc goes back almost 20 years), small access databases, as functions within Helpdesk systems, or recently everyone seems to have a need to write these in Sharepoint.  The mechanics of how you implement your process is not important, it's the fact that you implement a process that will make your changes successful !

If you have any thoughts, or if I've missed anything, please feel free to use the comment feature on this diary.  I'll update this article as the day goes on, based on your input.

=============== Rob VandenBrink Metafore ===============