skill-observabilitylisted

# Skill: Observability (Metrics, Logs, Traces) Three pillars: Metrics (what), Logs (why), Traces (how). Use all three. ## Metrics (Time-Series Data) **Key metrics**: - Request latency (p50, p95, p99) - Error rate (%) - Throughput (req/s) - Resource (CPU, memory, disk) **Collection**: Prometheus, StatsD, CloudWatch ``` api_latency_seconds{service="auth", endpoint="/login"} = 0.042 error_rate{service="api"} = 0.001 db_connections{pool="default"} = 42 ``` ## Logging (Structured) **Format**: JSON for parsing ```json { "timestamp": "2024-01-15T10:30:45Z", "level": "ERROR", "service": "user-service", "message": "Failed to create user", "error": "database_error", "user_id": "user-123", "duration_ms": 150 } ``` **Sampling**: Log 100% errors, 1% normal (reduce volume) ## Distributed Tracing **Trace request end-to-end**: ``` Request: POST /api/users ├─ Auth span (5ms) ├─ Validate span (2ms) ├─ DB insert span (45ms) │ ├─ Connection span (1ms) │ └─ Query span (44ms) └─ Cache update span (3ms) Total: 55ms ``` **Tools**: Jaeger, Zipkin, Datadog ## Dashboards (SLO-Focused) **What to display**: - SLO % (100% target, 99% actual) - Error budget consumed (%) - Latency (p95, p99) - Errors per minute - Active requests - Alert status **Tools**: Grafana, CloudWatch, Datadog ## Alerting **Good alert** (actionable): ``` Alert: DB connection pool > 80% Severity: CRITICAL Action: Increase pool size or check for connection leaks Runbook: https://wiki.example.com/alerts/d