sota-observabilitylisted

# SOTA Observability & Reliability ## Purpose Make every production system answerable. Two questions define success: 1. **"Why is this request slow/failing?"** — answerable for any single request from a trace ID, without adding new instrumentation. 2. **"What broke at 3am?"** — answerable from symptom-based alerts that page only when users are hurt, each linked to a runbook and a dashboard that narrows cause in minutes. This skill covers structured logging, metrics, distributed tracing, SLOs and alerting, and operational readiness — both how to **build** them correctly and how to **audit** them adversarially. Telemetry is a product with users (on-call engineers) and costs (storage, cardinality, attention). Treat both. ## BUILD mode When writing or modifying code, apply the rules files as design constraints, not afterthoughts. Workflow: 1. **Identify the signal need before coding.** For each new endpoint, job, or consumer: which SLI does it affect, what one wide event describes a unit of work, what spans bound its external calls. 2. **Instrument with OpenTelemetry API** (not vendor SDKs) in libraries; configure SDK/exporters only at the application entry point. Follow OTel semantic conventions for names and attributes. 3. **Emit one canonical wide event per request/job** at completion, carrying trace_id, outcome, durations, and business context. Debug logs are supplementary, sampled, and disposable. 4. **Propagate context everywhere**: W3C `tr