prometheus-troubleshootinglisted

# Prometheus Troubleshooting Establish whether the problem is collection, storage, query, or alerting — they look identical from a blank dashboard. ## Purpose **Use this when:** - A target is `DOWN` or missing from `/targets`. - A metric that used to exist returns no data. - A PromQL query returns nothing, or something implausible. - Prometheus is OOMing, restarting, or using far more memory than expected. - An alert did not fire during an incident, or fires constantly. **Do not use this when:** - Prometheus is fine and you are debugging the application it monitors — use `observability-debugging`. - You are designing alerts and SLOs rather than fixing them — use `slo-design`. **What this skill assumes:** access to the Prometheus UI or API, and ideally the scrape configuration. Many of these checks are answered by Prometheus's own metrics about itself. ## Operating Procedure 1. **Localise the layer.** Is the metric being scraped, stored, queried, or alerted on? Check in that order — each layer depends on the one before. 2. **Check the target first.** `/targets` and `up` answer most "missing metrics" reports in seconds. 3. **Query the raw series** before the aggregation. A broken `rate()` on a healthy counter is a query problem, not a data problem. 4. **Use Prometheus's own metrics** to diagnose Prometheus. `prometheus_tsdb_*` and `scrape_series_added` are the real evidence. 5. **Fix at the right layer.** A missing label is fixed in relabelling; a missing me