Logging & Observability
If you cannot diagnose it at 3 a.m., you did not finish the feature. Prefer high-cardinality-careful metrics, structured logs, and traces that connect them.
Three pillars (practical)
| Signal | Use for |
|---|---|
| Metrics | SLIs, alerts, capacity |
| Logs | rich context on discrete events |
| Traces | latency across services |
Workflow
- Define user journeys and SLIs (availability, latency, error rate).
- Identify golden signals per service.
- Specify structured log fields (request_id, tenant, error codes) — never secrets.
- Trace critical paths; propagation headers.
- Alert on symptoms (SLO burn) more than raw CPU when possible.
- Dashboard sketch: RED/USE + business KPI.
Output format
## Observability plan: <service>
### SLIs / SLOs
…
### Metrics
…
### Logs (fields & levels)
…
### Traces
…
### Alerts
…
### Privacy / cardinality risks
…
Rules
- Never log passwords, tokens, raw PII, or full payment payloads.
- Bound cardinality (no unbounded user IDs as metric labels).
- Correlate with a request/trace ID across services.
- Prefer logging errors with stable
error_codeover unique essay messages only. - Alerts need runbook links and human-actionable thresholds.
- Sampling traces is fine; always sample errors.
Edge cases
- Batch jobs: progress metrics, last success timestamp.
- Mobile/clients: privacy and volume constraints.
- Multi-tenant: tenant in logs/metrics with care for isolation and cardinality.