metrics-and-tracinglisted

# Metrics and tracing Three signals, three jobs. Confusing them produces expensive telemetry that answers nothing. | Signal | Answers | Cost model | | --- | --- | --- | | Metrics | *Is something wrong, and since when?* | Cheap, aggregated, bounded by cardinality | | Traces | *Where in the call path, and why slow?* | Per-request, sampled | | Logs | *What exactly happened to this one request?* | Expensive per event ([[structured-logging]]) | Start every investigation at metrics, narrow with traces, confirm with logs. If you instrument in the reverse order, you will pay the most for the least. ## What to measure **RED — for anything serving requests** (endpoint, consumer, RPC method): - **Rate** — requests per second - **Errors** — failed requests per second, as a ratio of the rate - **Duration** — latency distribution, as a histogram **USE — for resources** (pool, queue, disk, CPU): - **Utilization** — how busy - **Saturation** — how much queued work is waiting - **Errors** — rejections, timeouts Saturation is the one teams skip and the one that predicts outages. Connection pool wait time, queue depth, and worker-pool queue length tell you an hour ahead of the incident; utilization tells you during it. Instrument the **user-visible journey**, not only per-service internals — a checkout that fails because one of seven services is down is one broken journey, and the graph nobody has is the one that shows it. See [[slo-and-error-budgets]]. ## Metric types - **Counter**