Disaster Recovery
DR playbooks survive region loss, data corruption, ransomware, and operator delete. Not a single-alert procedure (runbook-writing), not live bridge command (incident-command), not availability targets alone (sla-slo-writing).
A backup is a copy. A restore is a working system. Untested is unverified.
Workflow
- Scope: systems, data classes, disaster classes (region, corruption, ransomware, vendor, operator delete).
- Inventory from evidence: method, location, encryption, retention, immutability, last verified success. Missing proof → UNVERIFIED.
- Objectives: declared RPO/RTO and who set them. Ask if unknown; never invent. Separate declared target from last measured restore time.
- Decision tree: fail over vs restore in place vs accept data loss. Who may declare DR.
- Procedures: verification, restore, failover, failback. One action per step. Mark overwrite / cutover danger.
- Dependencies: DNS, IdP, secrets, keys, network, vendors. Cannot decrypt or re-point DNS → failed restore.
- Drill: tabletop plus an actual restore. Success criteria and evidence. Never skip the restore test.
- Gaps: every unverified claim. First action is often a scoped drill.
Output format
# DR plan: <scope>
**Owner:** …
**Last reviewed:** …
**Last successful restore drill:** <date + evidence> | never
**Declared RPO / RTO:** … (source: …)
**Disaster classes:** …
## Inventory
| System | Data class | Backup | Location | Encryption | Retention | Immutable | Last verified |
## Objectives
| System | RPO | RTO | Who set it | Last measured restore |
## Decision tree
Fail over vs restore in place vs declare loss. Declarer / executor.
## Backup verification
How we prove the copy exists and is not empty/corrupt.
## Restore
1. …
## Failover / failback
1. …
## Ransomware / integrity
Isolate. Preserve forensic copy. Restore from known-good, not "latest".
## Dependencies & access
…
## Communications
Who, channel, customer facts (no invented ETAs).
## Drill plan
Cadence, scope, success criteria, evidence to retain.
## Gaps (unverified)
- …
## Related
runbooks / SLOs / owners
Rules
- Never invent restore times, backup success, or last-test dates. Ask or mark UNVERIFIED.
- Never recommend skipping restore tests. Paper drills are not a restore.
- Always treat RPO/RTO as requirements and measured restore as evidence.
- Always separate "backup exists" from "we restored a working system".
- Always name who declares DR and who may execute destructive failover.
- Mark overwrite, failback, and production cutover danger.
- Ransomware: isolate first; latest backup may be dirty; prefer immutable / offline known-good. Do not overwrite good copies.
- Single-alert steps →
runbook-writing. Live outage →incident-command. Targets-only wording →sla-slo-writing. - Missing evidence still gets a plan — Gaps stays red; no green checkmarks.
Edge cases
- Never restored: draft only; first action is a scoped restore drill.
- Ransomware / integrity: do not assume latest snapshot is clean.
- One AZ / partial region: failover may beat full restore; still need RPO.
- SaaS with no export: vendor RPO/RTO are claims; require their evidence.
- Keys died with the site: call out escrow; restore without keys fails.
- "Are we DR ready?" with no tests: refuse the claim; list what would make it true.
- Disaster is happening now: hand the bridge to
incident-command.