The situation
A media company runs about forty incidents a year of severity two or above. The response works: people are paged, they join a call, they fix things. What does not work is everything after.
The post-mortem is written from a chat channel, somebody’s notes and their recollection. The last three have taken more than a week to produce, two of them disagreed with the monitoring data about when impact started, and one attributed a cause that the metrics contradicted when somebody checked months later.
Three specifics. Nobody records when mitigation was applied, so mean time to restore is estimated. Actions from post-mortems are captured in the document and nowhere else, so completion is unknown. And the same contributing factor, a configuration change deployed without a canary, has appeared in four post-mortems in twelve months without anybody noticing until an annual review.
The requirement is a defensible timeline produced during the incident rather than after, without adding steps for the responder at three in the morning.
What actually matters
The first thing that matters is that anything requiring the responder to remember a step will not happen during a severe incident. The record has to be a by-product of the response rather than an additional task, which means the mechanism has to be attached to the things people already do: acknowledging a page, running a runbook, deploying a fix.
The second is that timestamps are what every later question rests on, and they are the first thing memory gets wrong. When impact started, when it was detected, when somebody acknowledged, when mitigation was applied, when it was confirmed resolved: those five give every duration anybody will ask for, and only the first is hard to capture automatically.
The third is that actions need to live where work lives. An action item in a post-mortem document is a good intention; the same item in the team’s tracker with an owner and a date is a commitment. The mechanism should create the second from the first.
The fourth is that the recurring factor is the whole reason to do this systematically. Four incidents sharing a cause is only visible if the incidents are structured records rather than prose, which means contributing factors need to be categorised rather than described.
Underneath it, the response itself is fine. This is entirely about the record, and conflating the two produces a proposal that adds process to a working response.
What we’ll filter on
- Does this capture happen automatically, or does somebody have to do it?
- Does it produce a timestamp, a decision, or an artefact?
- Is it available during the incident or only afterwards?
- Does it survive the chat retention policy?
- Can it be analysed across incidents, or only read one at a time?
- Does it add a step for the responder?
The landscape
Systems Manager Incident Manager. The purpose-built service: response plans, engagement with on-call contacts, escalation, an incident record with an automatically maintained timeline, and chat channel integration. Events are added to the timeline automatically as the incident progresses, and responders can add entries. It attaches runbooks to an incident, so executing one is recorded rather than remembered.
Automatic timeline events. Incident Manager records creation, engagement, runbook execution, property changes and resolution without anybody typing. This is what makes the record a by-product: acknowledging a page produces a timestamp.
Response plans with runbooks attached. An Automation runbook associated with a response plan is started as part of the incident, and its execution and result appear on the timeline. That covers the “when was mitigation applied” gap, because applying mitigation through the runbook records itself.
Chatbot and chat channels. Incident Manager can post to a Slack or Teams channel through Chatbot, so the channel people are already in becomes an interface to the incident record rather than a parallel universe of information that expires with the retention policy.
CloudWatch alarms creating incidents. An alarm configured to start an incident gives the detection timestamp directly from the alarm’s state change, rather than from when a human noticed. This is the second of the five timestamps, captured for free.
Post-incident analysis in Incident Manager. A structured analysis attached to the incident, with questions, findings and action items, which keeps the analysis with the record rather than in a separate document.
EventBridge from incident lifecycle. Incident Manager emits events, so an action item can create a ticket in the team’s tracker automatically, which is what moves actions from a document into the work queue.
Structured contributing factors. Categorising causes rather than describing them, so “configuration change without a canary” is a value that can be counted across forty incidents rather than a sentence in four documents.
Application Signals and service level objectives. Where an SLO exists, the burn is the impact-start timestamp, which is the one timestamp humans consistently get wrong because the first report arrives well after the impact began.
Evaluation
Side by side
| Mechanism | Captures | Automatic | Adds a responder step | Analysable across incidents |
|---|---|---|---|---|
| Chat channel scrollback | Discussion | ✓ | ✗ | ✗ |
| Manual notes | Whatever is remembered | ✗ | ✓ | ✗ |
| Alarm starting an incident | Detection time | ✓ | ✗ | ✓ |
| Incident Manager timeline | Engagement, runbooks, resolution | ✓ | ✗ | ✓ |
| Runbook attached to a response plan | Mitigation time and what was done | ✓ | ✗ | ✓ |
| SLO burn | Impact start | ✓ | ✗ | ✓ |
| Post-incident analysis | Findings and actions | ✗ | Afterwards | ✓ if structured |
| EventBridge to a tracker | Action items as work | ✓ | ✗ | ✓ |
Four of the five timestamps come out of the top half with no responder involvement at all. The one that does not, impact start, comes from the SLO burn rather than from anyone’s recollection, which is the row that fixes the disagreement between the post-mortems and the monitoring data.
The solution
Wire the incident record to the alarms and the runbooks so four of the five timestamps are free, take impact start from the SLO, and push action items into the tracker automatically.
Start with response plans in Incident Manager, one per service, with the engagement plan matching the existing on-call rotation. Nothing about who gets paged changes; the difference is that the paging now produces an incident record with a timeline.
Configure the alarms that currently page to start an incident instead. The alarm’s state change becomes the detection timestamp, engagement produces the acknowledgement timestamp, and both are recorded without anybody typing. This is the change that removes the two most-disputed numbers from memory.
Attach the existing Automation runbooks to the response plans, so running one during an incident is an action on the incident rather than a console session somebody had. The execution start and result land on the timeline, which is where “when was mitigation applied” comes from. This is also the step that makes runbooks more likely to be used, because they are presented in the response rather than looked up.
Connect the chat channel through Chatbot, so the channel the team already uses shows the incident and lets responders add timeline entries without leaving it. The discussion still happens in chat; the record no longer depends on chat retention.
Take impact start from the service level objective’s burn rather than from a human. Where an SLO exists, the moment the error budget started burning is the impact start, and it is defensible in a way “I think it began around ten past” is not. Where no SLO exists, that is the gap to close for that service, and it is worth knowing which services those are.
Then structure the analysis. Use the post-incident analysis attached to the incident rather than a separate document, and add a categorised contributing-factor field with a fixed vocabulary. Four incidents sharing “configuration change without a canary” becomes a count on a dashboard rather than a pattern somebody spots at an annual review.
Push the actions out. An EventBridge rule on the analysis creating tickets in the team’s tracker, with the owner and the incident reference, means an action item is work rather than a line in a document nobody reopens.
Then measure the record itself. Time from resolution to completed analysis, and the proportion of action items closed, are the two numbers that say whether this is working. A week to produce a post-mortem was the symptom; if it does not fall, the mechanism has added process without adding value.
Why not just require better note-taking during incidents. It is what the team is already doing and it depends on somebody having spare attention at three in the morning, which is exactly when there is none.
Why not build this on the existing chat channel with a bot. It captures the discussion, which is the part already captured, and it inherits the chat retention policy and produces prose rather than structured records that can be counted across forty incidents.
Worked example
The response plans go in over two weeks, one service at a time, with no change to the rotation. The first incident under the new arrangement produces a timeline nobody had to write, and the responder’s feedback is that they did not notice a difference during the response, which is the intended experience.
The impact-start change produces the first real finding. For three incidents in the first two months, the SLO burn shows impact beginning between 4 and 19 minutes before the alarm fired. Under the old process those minutes were invisible, and mean time to restore had been understated by roughly a fifth for years.
Attaching runbooks has an unexpected effect on usage. Runbook executions during incidents roughly double in the first quarter, because they are presented in the incident rather than found in a wiki, and the mitigation timestamps come for free as a consequence.
The categorised contributing factors take an argument to agree, because everyone wants a free-text field. The compromise is a fixed vocabulary of twelve plus a free-text note. Within a quarter the dashboard shows “configuration change without a canary” at five occurrences, which turns a pattern nobody had counted into a funded piece of work on the deployment pipeline.
Action items in the tracker close at about 70% within a month, against an unknown baseline that everyone suspected was much lower.
Time from resolution to completed analysis falls from over a week to two days, mostly because the timeline no longer has to be reconstructed and the disagreements about timestamps stopped happening.
What’s worth remembering
- Anything requiring the responder to remember a step will not happen during a severe incident, so the record has to be a by-product of the response rather than an additional task.
- Five timestamps answer every duration anybody asks for, and four of them (detection, engagement, mitigation, resolution) come free from alarms, engagement and runbook execution.
- Impact start is the one humans consistently get wrong, and the SLO’s error-budget burn is the defensible source for it.
- Attach runbooks to response plans: the execution is recorded automatically, and presenting them in the incident makes them more likely to be used at all.
- Categorise contributing factors with a fixed vocabulary, because a recurring cause across four incidents is only visible if the records can be counted rather than read.
- Push action items into the team’s tracker automatically; an action in a post-mortem document is an intention, and the same item with an owner and a date in the work queue is a commitment.