Incident Management & Blameless Postmortems

Detect, respond, and learn fast enough that reliability compounds

Failure is inevitable at scale, so the differentiator is not preventing every incident but detecting, responding, and learning fast enough that reliability compounds over time. Incident management is the structured response; the blameless postmortem is the learning loop. Companies usually adopt both right after their first big outage teaches them that ad-hoc heroics do not scale.

Severity levels

Severity levels give everyone a shared vocabulary for "how bad" and the expected response. A typical scheme: SEV1 = critical, major user-facing outage or data loss, all-hands, wake people up; SEV2 = significant degradation, urgent but not everything-down; SEV3 = minor/partial, handled in business hours; SEV4 = negligible, tracked but not paged. Each level has explicit entry criteria (e.g. "checkout error rate > 5% for 5 min = SEV1") so declaring severity is objective, not a debate, and the severity drives who is paged and how often you communicate.

Roles separate coordination from fixing

Roles exist to separate coordination from fixing, because the person elbow-deep in the database should not also be fielding "is it fixed yet?" from executives. The Incident Commander (IC) owns the response, makes decisions, and delegates; they do not fix, they coordinate. The Communications Lead posts regular updates to stakeholders and the status page on a fixed cadence. The Operations/Scribe does the hands-on remediation and keeps a timestamped log of actions. This is adapted from emergency-services incident command. In a small SEV3 one person may hold several hats; in a SEV1 they are distinct people.

Mitigate before you diagnose

The response flow prioritizes mitigation over diagnosis: detect -> triage/declare severity -> mitigate (stop the bleeding) -> then root-cause. During an active incident, restoring service beats understanding it. If a bad deploy is suspected, roll it back first and investigate after; if a region is unhealthy, fail traffic away first. Chasing the root cause while users are down is a classic and expensive mistake. Diagnosis is what the postmortem is for; mitigation is what the incident is for.

detect -> declare SEV + assign IC/Comms/Ops -> MITIGATE (rollback / failover / shed) -> service restored
        -> (only now) diagnose root cause -> blameless postmortem -> action items -> tracked to done

The blameless postmortem

The blameless postmortem is the compounding step. Blameless means it assumes everyone acted reasonably with the information and tools they had, so the analysis targets the system that allowed the failure, not the individual who tripped it. Structure: a timeline (what happened, when), impact (users/revenue/duration), contributing causes (usually several, not one), and action items with named owners and due dates. Action items that are not tracked to completion are the reason the same incident recurs.

Interview nuance: never accept "human error" as a root cause, and be ready to say why. "Engineer ran the wrong command" is a stopping point that hides the real questions: why did the system let a single command take prod down, why was there no confirmation or dry-run, why did no guardrail catch it? Ask "why did the system allow this?" (this is the spirit of the Five Whys). Blameful postmortems make people hide mistakes, which hides the next incident, so blamelessness is a reliability strategy, not just a kindness.

Recap: objective severity levels drive the response, separate the Commander from the fixers, mitigate before you diagnose, and run a blameless postmortem that targets the system (never "human error") with owned action items tracked to completion.