loom-prometheus

# Prometheus Monitoring and Alerting Pull-based, multi-dimensional time-series monitoring with PromQL. Scope: PromQL, scrape/SD config, recording & alerting rules, instrumentation, Alertmanager. **Dashboards/panels → loom-grafana; log queries → loom-logging-observability.** ## Overview | Component | Role | | --------- | ---- | | Prometheus server | Scrapes targets, stores TSDB locally, evaluates rules | | Alertmanager | Dedup, group, route, inhibit, silence, notify | | Exporters | Translate third-party systems to the exposition format (node, blackbox, ...) | | Client libraries | In-process instrumentation (Go, Python, Rust, Java, ...) | | Pushgateway | Ephemeral batch jobs push metrics — see gotcha; avoid for services | | Operator / Thanos·Mimir·Cortex | K8s CRD deploy / long-term + global-view remote storage | **Metric types.** `up` is the synthetic per-target health series (1 = scrape ok). | Type | Semantics | Query with | | ---- | --------- | ---------- | | Counter | Monotonic, resets to 0 on restart (`_total` suffix) | `rate()`/`increase()` — **never graph raw** | | Gauge | Up/down (memory, queue depth) | raw, `avg_over_time`, `deriv`, `predict_linear` | | Histogram | Bucketed observations → `_bucket{le}`,`_sum`,`_count` | `histogram_quantile()`; buckets are additive | | Summary | Client-side quantiles → `{quantile}`,`_sum`,`_count` | **not aggregatable** — single-process only | ## Scrape Configuration ```yaml # prometheus.yml global: scrape_interval: 15s ...