In a lot of ways, our job in IT and Information Security is implementing change. But as we all know, every change involves risk, and changes gone bad can be your worst nightmare. I’ve seen the number of business system service interruptions due to changes in infrastructure pegged at anywhere from 50 to 70%, and I’d lean towards the high side of that. If it’s not a hardware failure, it’s probably a mistake. Why is this number so high?' Testing takes time, time that we often don't have. Managers will often consider every change to be "simple", so getting testing time is often a challenge. We'll often be working in complex environments, environments that involve lots of people who don't care about your change until it breaks something of theirs. We often have system components in the mix that we don't know about until something breaks. We may have legacy apps that simple cannot be known - stuff that was written 20 years ago by people who have long since retired for instance.. Compound this with the pervasiveness of the GUI interface , which in a lot of ways encourages people to stumble through the menu until they find a setting that works (yes this happens, every day). • Figure out what you’re doing You can do all of this with a few emails, but as I mentioned, most of us have forms to make sure we don’t forget steps in this, and we have meetings to make sure that everyone who needs to know gets told (and doesn’t have the “but nobody told me” excuse later). What is a change? Especially when change control is new in an organization, the topic of what change is will be a hot one - when do you need to fill out a change control request? I've been in organizations where unplugging an unused ethernet cable is a change, or plugging in a power cable. On the face of it, you might think that these are both going overboard. But if you unplug that dead cable you may find out later that it's not really dead - it's only used to run that legacy accounting report at month end. Or when you plug in the new IPS unit you may trip the breaker on the PDU that the main firewall or fileserver is plugged in to. The level of change that needs to hit the process will vary from company to company, if you're just starting out in this, no worries, just be prepared for some healthy back-and-forth as you find the level between "too much paper work" and "enough process to get the job done" The meeting before the meeting Change control meetings should be SHORT. I normally see discussion limited to 5 minutes per change – discuss any last minute details, be sure that change A and Change B don’t conflict, then it’s a “yes” or a “no”. If you go over 5 minutes, either you didn’t do your homework (remember the meeting before the meeting?), or someone is hijacking your meeting. Speaking of hijacking, change control meetings need a chair. In many companies, the chair rotates between everyone who is NOT a manager. This is a good thing for the team, as it gives people who otherwise wouldn’t speak during a meeting practice in talking to actual people. It also emphasizes that the meeting is to make the job of the "doers" easier - at its root change control is not supposed to be a heirarchal management function at all. On the other hand, in lots of other companies the IT manager chairs the meetings. In either case, the main job of the meeting chair is to keep it moving, to be sure that changes are approved or nixed quickly so everyone can get back to work. Finally, any completed changes should be discussed. The point of this is to continually make the process better. Reviewing completed changes is all about making sure that the testing process is good, and that people are writing good implementation and backout plans. Doing the Deed
I wrote this diary because I end up talking to one client or other about change control processes a few times per month - it's a pretty popular topic (especially when you're convincing people who don't have a process that they need one). I've attached a a sample change control form below. If you find them useful, feel free to modify this and use it in your organization (or not). Over the years, I've seen change control implemented as Word documents in a shared folder or paper binder (this doc goes back almost 20 years), small access databases, as functions within Helpdesk systems, or recently everyone seems to have a need to write these in Sharepoint. The mechanics of how you implement your process is not important, it's the fact that you implement a process that will make your changes successful !
If you have any thoughts, or if I've missed anything, please feel free to use the comment feature on this diary. I'll update this article as the day goes on, based on your input.
=============== Rob VandenBrink Metafore =============== |
Rob VandenBrink 578 Posts ISC Handler Aug 19th 2010 |
Thread locked Subscribe |
Aug 19th 2010 1 decade ago |
change is good, but you go first.
|
Ken 40 Posts |
Quote |
Aug 19th 2010 1 decade ago |
> I'll update this article as the day goes on,
> based on your input. Make sure you go through the proper process and procedures before making any changes to this article. |
Ken 7 Posts |
Quote |
Aug 19th 2010 1 decade ago |
You forgot the part about the meeting after the meeting, to discuss what went good and bad during the actual meeting, and to schedule a follow-up meeting.
Oh, and you'll always have to deal with the people who are against the change, and their only reason, which cannot be argued against, is that "This is the way it has always been done". |
Ken B 4 Posts |
Quote |
Aug 19th 2010 1 decade ago |
Maybe we should have a follow on poll - How many folks have good change management policies and processes. I'd love to see (and learn from) how other folks manage their operations.
|
Dean 135 Posts |
Quote |
Aug 19th 2010 1 decade ago |
Maybe we should have a follow on poll - How many folks have good change management policies and processes. I'd love to see (and learn from) how other folks manage their operations.
|
Dean 135 Posts |
Quote |
Aug 19th 2010 1 decade ago |
A prominent security professional once said, "Planning is for those who don't know how to troubleshoot." :)
One of the things that I've learned to do (at least on the larger/more visible work) is to map out all of the things that have to happen, estimate duration of each step, and get it all down on paper. Figure out what can be done ahead of time to prepare and make sure that it's on the document. For us, the change window starts when the changes actually begin. Sometimes the prep may be an hour leading up to it, but the prep is not the change itself. A recent project involved migrating firewalls to new hardware. I had it planned out in a roughly 4.25-hour process requiring about 90 steps. I had a time attached to each one, so I knew when I was four minutes behind or two minutes ahead. Steps were marked as mandatory or optional so that we could get caught up if anything went too long and we risked the 5-hour mark. We ended up finishing 35 minutes before the end of the window, seven minutes later than planned, but only because something unrelated that was to be finished early ate fifteen minutes into the beginning of our window. It's not something that I will do every time. Most of our outages are a reboot or a service restart, and I'm not going to document a two-step process as above. Those are easy to estimate, and if an hour-long window is set aside, you're covered. But for larger ones, the prep is necessary. |
Jarrod 5 Posts |
Quote |
Aug 19th 2010 1 decade ago |
No change happens without a security assessment.
|
Jarrod 7 Posts |
Quote |
Aug 19th 2010 1 decade ago |
tldnr ... too little time!
|
Andrew 41 Posts |
Quote |
Aug 20th 2010 1 decade ago |
And there's the crux of our problem. Do we not have time or do we not see the value in the extra steps. I'm wondering how many of the folks that don't do this also don't produce metrics or have any type of service level agreement.
|
Dean 135 Posts |
Quote |
Aug 20th 2010 1 decade ago |
Whilst most people scream when you say this if they've ever had any thing to do with it, I'm going to mention it anyway: ITIL.
But not the full blown thing - that's suicide for most people, remember that there is a supplement called "Small-Scale Implementation". This is quite useful as a basic intro if you're not a huge organisation, and can be quite useful to have a good look at, even if you don't implement it formally (and it's far to often that people implement the whole thing rigidly, and it basically kills your operation). I also take exception that their 'best practices' are best for everyone, but again, it's worth a read. The "Service Transition" bit is basically the change part. |
Alex 19 Posts |
Quote |
Aug 20th 2010 1 decade ago |
After reading "The Visible Ops Handbook" (highly recommended), we've used a simple change management system for several years, which I describe here:
http://rockssandwater.blogspot.com/2010/01/simple-change-management-system-pays.html I would never go back to the old ways. |
Alex 8 Posts |
Quote |
Aug 21st 2010 1 decade ago |
Sign Up for Free or Log In to start participating in the conversation!