One of the things that comes ups frequently in discussion is the difference between incident response, and incident handling.  Anyone who has had to deal with an incident has likely encountered this situation at least once.  While attempting to work on analyzing what the #$%& happened you always have your senior executive and/or clients hanging over your shoulder constantly bugging you for more details. The containment plan needs to be worked out, someone needs to liaise with Legal/HR/PR, management wants an update, the technical staff need direction or assistance, teams need to be coordinated, everyone wants to be in the loop, lots of yelling is going on, external IRTs want to know why your network is attacking theirs, nobody can locate the backups, keeping track of activities, taking notes, and the list goes on… No wonder people regularly burn out during incidents! Incident handling is obviously not a solo sport.

The best line for this problem came from the hot wash (post incident debrief) of a major exercise, where one of the leads stood up to the table of senior executives, and calmly explained "No, no you will NOT go talking to the team.  You will talk to me. Only me.  It is my job to keep you off their backs so they can do actual work".

That is the difference between Incident Response, and Incident Handling.  Incident Response is all of the technical components required in order to analyze and contain an incident.  Incident Handling is the logistics, communications, coordination, and planning functions needed in order to resolve an incident in a calm and efficient manner. Yes, there are people who can fulfill either role, but typically not at the same time. The worse things get, the greater the requirement for the two different roles becomes.

These two functions are best performed by at least two different people, although in larger scale incidents this may need to be two leads who work closely together to coordinate the activities of two separate teams. In smaller environments ideally the Incident team should still always be two people, one responding, and the other taking notes and communicating with stakeholders.

It should also be noted that the skill sets for these two are different as well.  Incident Response requires strong networking, log analysis, and forensics skills; incident handling strong communications and project management skills.  These are complementary roles which allow the responders to respond, the team to work in a planned (or at least organized chaos) fashion and the rest of the world to feel that they have enough information to leave the team alone to work.

Thanks to a former colleague who helped me writing this.

Thoughts or feedback?

Update: Rex wrote in the following: US workers in emergency services (fire, police, ambulance, first aid), are trained in the Incident Command System:  http://en.wikipedia.org/wiki/Incident_Command_System
which defines a number of important roles in handling an emergency incident.

Small teams won't have enough members to put one person in each role.  Experience shows that the "hands-on" members *must be left alone*, even if you need to do on-the-spot recruitment to handle PR, crowd control, logistics, etc.  Another key concept is "one person in charge per function" - one commander, one PR person, one chief medic, etc.

Maybe the IT security incident response/handling world should steal good ideas from people who deal with life-and-death emergencies.  Soon, IT emergencies will be life-and-death emergencies.

Cheers,
Adrien de Beaupré
Intru-shun.ca Inc.