Incident management: what to do when everything breaks

Incident management is an agreed procedure for when something breaks: who learns about the problem, who coordinates, how users are informed, how service is restored and how to make sure it does not happen again. The first principle is to restore service first and look for the root cause later. The second is to review incidents without blaming people.

Here are severity levels, incident roles, communication and postmortems.

What counts as an incident?

An incident is any event that disrupts or threatens the product for users: the site is down, payments fail, data leaks, errors spike. To respond proportionately, incidents are classified by severity.

Level	Example	Response
SEV1 — critical	Site down, payments failing, data breach	Immediately, any time, everyone needed
SEV2 — high	An important feature broken for some users	Within an hour, on-call and the owning team
SEV3 — medium	A bug with a workaround, slowdown	During working hours
SEV4 — low	Cosmetic issues	As usual, as a backlog task

On learning about outages in time, see monitoring and logging.

Which roles are needed during an incident?

During a serious incident one person should coordinate, rather than everyone fixing, messaging customers and arguing in a shared chat at once. Even in a small team it helps to assign roles explicitly.

• incident lead — makes decisions, assigns tasks, watches the clock and does not fix things personally;
• responders — diagnose and resolve the problem;
• communications owner — updates users, support and management;
• scribe — records the timeline of actions and decisions for the review.

In a small team one person can combine lead and communications, but responders should stay focused on the problem.

Photo: Unsplash

How do you act during an incident?

The first goal is to restore service as fast as possible, even with a temporary fix: roll back a release, switch off a problematic feature, fail over to a standby server. The root cause can wait for the review.

Steps:

1. Confirm and assess. What broke, for whom, how badly — assign a severity.
2. Assemble people. Open a dedicated incident channel and name a lead.
3. Communicate. A short first message to users and support: we know, we are working on it.
4. Stabilise. Roll back, disable the feature with a flag, fail over — see feature flags and zero-downtime deployments.
5. Update status. Regularly, even when there is no news.
6. Confirm recovery. By metrics and checks of key flows, not by feel.

What is a blameless postmortem?

A postmortem is a review after recovery: what happened, why, how we responded and what to change. Blameless means looking for weaknesses in systems and processes rather than a person to punish. Otherwise people start hiding mistakes and the next incident repeats.

A postmortem includes:

• a short summary and impact: how many users, how long, what was lost;
• a timeline: when it started, when it was noticed, what was done, when it was resolved;
• causes — usually several, not one;
• what went well and what got in the way;
• concrete actions with owners and deadlines.

How do you stop incidents from recurring?

Incidents recur when postmortem actions stay in the document and never reach the work plan. Actions must be concrete, owned and dated, and done with the same priority as product work.

• add monitoring and alerts that would have caught the problem earlier;
• automate manual steps that slowed recovery;
• add tests and CI checks — see automated testing strategy;
• update recovery runbooks and test them in drills;
• track time to detect and time to recover, and watch them fall.

On availability commitments, see SLA, SLO and SLI.

In short

• Incidents are classified by severity and handled proportionately.
• An incident needs a lead, responders and a communications owner.
• Restore service first, find the root cause later.
• A postmortem looks for system weaknesses, not a person to blame.
• Follow-up actions get owners, deadlines and real priority.