observability-stacklisted

# Observability Stack ## Overview Production observability using the three pillars (metrics, logs, traces) with OpenTelemetry as the collection standard. Covers metric design, alerting strategies, dashboard patterns, and cost management. ## The Three Pillars | Pillar | What | Tool Stack | When | |--------|------|-----------|------| | **Metrics** | Numeric time-series data | Prometheus + Grafana | Trend detection, alerting, capacity | | **Logs** | Structured event records | Loki / ELK / CloudWatch | Debugging, audit trails, forensics | | **Traces** | Request flow across services | Tempo / Jaeger / X-Ray | Latency analysis, dependency mapping | All three must be **correlated** via trace context (trace_id, span_id) for effective debugging. ## Metric Design ### RED Method (Service-Level) Measure what **users experience**. Apply to every service endpoint. ```promql # Rate — requests per second rate(http_requests_total{service="api"}[5m]) # Errors — error percentage rate(http_requests_total{service="api", status=~"5.."}[5m]) / rate(http_requests_total{service="api"}[5m]) * 100 # Duration — p99 latency histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="api"}[5m])) ``` ### USE Method (Infrastructure-Level) Measure what the **infrastructure is doing**. Apply to every resource. | Resource | Utilization | Saturation | Errors | |----------|------------|------------|--------| | **CPU** | `node_cpu_seconds_total` | `node_load15` / num_cpus | — | | **M