What you'll learn
Quick Answer
Restore service first, understand it afterwards. Roll back rather than debugging forward, communicate early even without answers, and write a blameless postmortem focused on the system rather than the person.
Stop the bleeding before diagnosing
The most common mistake during an outage is spending twenty minutes understanding the root cause while users cannot use the product.
Mitigation and diagnosis are separate activities, and mitigation comes first.
If a deployment went out in the last hour, roll it back. Do not investigate what in it broke — roll back, confirm service is restored, and investigate afterwards with the pressure off. The change is still in git; nothing is lost by reverting.
Other fast mitigations worth reaching for: disable the feature behind a flag, scale up if it is load-related, restart the affected service, or route traffic away from a failing instance.
None of these fix the underlying problem, and that is fine. Users care that it works, not that you understood it quickly.
Assess before acting
Sixty seconds of assessment prevents a panicked action that makes things worse.
- What is the user-visible impact? Everyone, or one feature? Failing entirely, or slow?
- When did it start? Correlate with deployments, configuration changes and scheduled jobs. Most incidents begin with a change.
- What changed? The single highest-value question. Deployments, config, feature flags, infrastructure, a dependency, or a certificate that expired.
- Is it getting worse? A queue backing up or a disk filling has a deadline attached.
In a team, name an incident lead immediately. Their job is coordination and communication, not fixing. Without one, three people independently restart the same service, and two of them do not know the third already tried it.
Communicate before you have answers
The instinct is to wait until you understand the problem. That is backwards — silence is worse than uncertainty, because users assume nobody has noticed.
An early message costs nothing and buys patience:
"We are aware that checkout is failing for some users. We are investigating and will update in 30 minutes."
No cause, no promise, and a commitment to update. Then actually update at that time, even if the update is "still investigating, next update in 30 minutes".
Three rules: do not speculate publicly about causes you have not confirmed; do not promise a fix time you cannot know; and say when it is resolved, since people are still waiting.
Internally, keep a running timeline in one channel as you go. Reconstructing it afterwards from memory is unreliable, and the postmortem depends on it.
Diagnosing under pressure
Once service is restored, or while mitigation is running, the ordinary debugging discipline applies — with one addition.
Change one thing at a time. Under stress people apply three fixes simultaneously, service recovers, and nobody knows which worked or whether the other two introduced new problems.
Start with what changed, because most incidents are caused by a change. Then work through the request path from the outside in: DNS, load balancer, application, database, dependencies. Check the obvious physical things — disk space, certificate expiry, connection pool exhaustion — because they cause a surprising share of outages and are quick to rule out.
Your monitoring should answer most of this. If it cannot, note that as an action item; "we could not tell what was happening" is itself a finding.
And know when to stop. If you have been at it for hours with no progress, escalate or hand over. Tired debugging causes second incidents.
The postmortem, and why blameless matters
After resolution, write it up. Not to assign fault — to stop it recurring.
A useful postmortem contains a timeline, the user impact, what caused it, what made it worse or better, and specific action items with owners.
Blameless is not politeness, it is accuracy. If people expect blame, they minimise their involvement and the timeline becomes incomplete — so you fix the wrong thing. Culturally, the finding is never "Ravi deployed a bad config"; it is "a config change reached production without validation, and there was no automated check that would have caught it".
The second version produces an action item. The first produces a person who is careful for two weeks.
Two practical points. Ask "why" repeatedly until you reach a systemic cause rather than a human one. And keep the action list short and owned — a postmortem with fifteen unassigned items produces nothing, while two completed ones genuinely prevent recurrence.