Runbook Writing
Runbooks are for 3 a.m. brains. Short steps, copy-pasteable commands, clear stop conditions, and escalation paths.
Workflow
- Scope: service, symptoms/alerts that trigger this runbook, severity.
- Preconditions: access, dashboards, feature flags, dependencies.
- Diagnosis steps with expected outputs.
- Mitigation steps (customer impact first) then durable fix pointers.
- Rollback / communications / escalation.
- Post-incident: what to capture for the postmortem.
Output format
# Runbook: <title>
**Service:** …
**Alert / symptom:** …
**Severity guidance:** …
**Last reviewed:** …
## Overview
One paragraph.
## Dashboards & signals
- …
## Diagnosis
1. …
## Mitigation
1. …
## Escalation
- …
## Rollback
…
## Do not
- …
## Related
links / owners / code paths
Rules
- Imperative steps; one action per step.
- Commands in copy-paste blocks with placeholders like
<cluster>. - Never require tribal knowledge without a link or command to discover it.
- Separate customer mitigation from root-cause engineering.
- Include "when to stop and escalate".
- Mark destructive steps with danger and confirmation.
Edge cases
- Partial outage: branch the path by symptom.
- Security incident: avoid spreading exploit detail; point to secure channel.
- No prod access in doc context: still structure the runbook; mark missing commands clearly.