loom-logging-observability

# Logging and Observability ## Overview Understand system behavior through the three pillars — logs, metrics, traces — correlated by shared IDs. This skill covers structured logging, OpenTelemetry tracing, Prometheus metrics, aggregation backends, and alerting, with emphasis on the cost/cardinality traps and sampling decisions that separate a working setup from an expensive broken one. ## Three Pillars — what each answers, and its cost model | Pillar | Answers | Cost driver | Use for | | ----------- | ------------------------------------ | ----------------------------------- | ---------------------------------------------- | | **Metrics** | "Is it broken? how much?" (aggregate) | Label **cardinality** (# series) | Dashboards, SLOs, alerting — always-on, cheap | | **Traces** | "Where in the request path?" (causal) | Span volume → **sampling** | Latency breakdown, cross-service dependency | | **Logs** | "What exactly happened?" (event detail) | Volume + **indexing** strategy | Forensics, audit, the specifics of one request | Reach for metrics first (cheap, aggregate), traces to localize, logs for the detail. Link all three by `trace_id`/`correlation_id` so you can pivot: alert fires on a metric → jump to an exemplar trace → read that trace's logs. ## Structured Logging Emit JSON, one object per event — never string-interpolated prose. Structured field...