Chaos Experiment Design
Chaos without a hypothesis is vandalism. Scope blast radius, define steady state, abort criteria, and learnings — prefer staging before prod.
Workflow
- Hypothesis: if X fails, Y still holds (SLO/user journey).
- Steady-state metrics and dashboards.
- Fault: type, target, magnitude, duration.
- Blast radius, blast radius controls, abort conditions.
- Observability and comms (who knows, status page if needed).
- Runbook for inject + stop; post-experiment notes.
- Never surprise prod; approval for prod experiments.
Output format
## Experiment: <name>
**Hypothesis:** …
**Steady state:** …
**Method:** …
**Blast radius:** …
**Abort if:** …
**Schedule / env:** …
**Rollback:** …
**Learning questions:** …
Rules
- No unannounced production chaos.
- One variable at a time when possible.
- Customer-impacting faults need explicit approval language.
- Capture learning even if the system “passed”.
- Do not invent tool-specific syntax you are unsure of.
Edge cases
- Game days: facilitation roles + scenario script.
- Multi-region: partial vs full region loss.
- Data plane faults: disk full, poison messages, clock skew.