observability-setup

# Observability Setup ## Overview Observability is the ability to ask arbitrary questions about a system's behavior from the outside, using its telemetry. This skill instruments a service across the **three pillars** — structured logs, metrics, and distributed traces — then turns raw signals into **SLIs**, **SLOs**, **error budgets**, and **actionable alerts**. The goal is not "more dashboards" but the ability to answer *"is the user in pain, and where?"* in under five minutes. **Keywords:** observability, OpenTelemetry, OTel, structured logging, JSON logs, Prometheus, metrics, RED method, USE method, distributed tracing, spans, trace context, correlation ID, SLI, SLO, error budget, burn rate, alerting, Alertmanager, Grafana, on-call, golden signals, cardinality, exemplars. Standardize on **OpenTelemetry (OTel)** for instrumentation — it is vendor-neutral and emits to Prometheus, Tempo, Loki, Datadog, Honeycomb, etc. Keep instrumentation code free of vendor SDKs; push vendor choice to the OTel Collector. ## Workflow 1. **Inventory the service.** Identify language/runtime, request entry points (HTTP/gRPC/queue), critical user journeys, downstream dependencies, and the existing telemetry backend (or pick one). Read `references/instrumentation-patterns.md` for per-pillar guidance. 2. **Add structured logging first.** Emit one JSON event per request with a stable schema (timestamp, level, message, `trace_id`, `span_id`, service, plus event-specific fields). Never log secret...