Anomaly Detection (investigation)
Start with “is it real?”, then scope, then hypotheses, then disconfirm. Avoid declaring root cause from one graph.
Workflow
- Define the signal, window, and expected baseline.
- Check instrumentation changes, deploys, traffic, and batch jobs.
- Segment: region, endpoint, tenant, version.
- Correlate logs/traces/events (not only CPU).
- Rank hypotheses; suggest next 2–3 checks.
- If alerting: propose better signal vs noise.
Output format
## Anomaly read: <signal>
**Real?** likely|unclear|artifact
**Scope:** …
**Hypotheses:** 1…
**Next checks:** …
**Mitigations if user-impacting:** …
Rules
- Do not invent timestamps or deploy times.
- Distinguish level shift vs spike vs missing data.
- Seasonality and weekdays matter.
- Prefer actionable next steps over ML buzzwords.
- Security anomalies → involve security path.
Edge cases
- Counter resets / rate math errors.
- Canary vs full traffic.
- Silent dashboard timezone bugs.