implementing-pipeline-observabilitylisted

# Implementing Pipeline Observability ## When to use - Failures or stale data are discovered by stakeholders, not by your alerts. - Setting up monitoring, SLAs, or on-call for data pipelines. - Alerts are too noisy (fatigue) or too quiet (silent failures). - Do NOT use for one-off incident debugging (use `debugging-data-pipelines`). ## The four signals to monitor 1. **Freshness** — is data arriving within its SLA? (most incidents are lateness) 2. **Volume** — is row count within the expected range vs a trailing baseline? 3. **Quality** — do the data quality checks pass? (see `implementing-data-quality-checks`) 4. **Run health** — job success/failure, duration, and retry rate. ## Workflow ``` - [ ] Define SLAs per critical dataset (freshness + volume) - [ ] Emit run metrics: status, duration, rows in/out per run - [ ] Add freshness + volume monitors with baseline-relative thresholds - [ ] Route alerts to an owner with context and a runbook link - [ ] Tune thresholds to cut false positives; track MTTD/MTTR ``` 1. **Set SLAs** for datasets that feed decisions ("orders fresh within 3h, ±20% daily volume"). Without an SLA there is nothing to alert against. 2. **Emit run metrics** (status, duration, input/output row counts) to a store you can query and chart, not just scattered logs. 3. **Monitor freshness and volume** against **trailing baselines**, not fixed numbers, so seasonality doesn't trigger noise. 4. **Make alerts actionable** — every alert names the dat