managing-data-lineage-openlineagelisted

# Managing Data Lineage with OpenLineage ## When to use - Setting up automated lineage across Airflow, dbt, and Spark. - Answering "what breaks if I change this table?" (impact analysis). - Scoping an incident's blast radius or a backfill's downstream footprint. - Do NOT use for in-project dbt lineage only (dbt docs cover that; use this for cross-tool, cross-system lineage). ## Core model OpenLineage describes each pipeline run as **Job** (the transformation), **Run** (one execution), and **Datasets** (inputs/outputs), optionally with **column-level** mappings. Emitters send these events to a backend (e.g. Marquez) that builds the graph. ## Workflow ``` - [ ] Add the OpenLineage integration to each producer (Airflow/dbt/Spark) - [ ] Point emitters at a collector (Marquez or vendor endpoint) - [ ] Verify jobs/datasets/edges appear with column-level detail where supported - [ ] Use the graph for impact analysis before changes - [ ] Use it during incidents to scope downstream impact and backfills ``` 1. **Instrument producers.** Airflow (OpenLineage provider), dbt (`dbt-ol`/native), and Spark (OpenLineage listener) emit events automatically — prefer these over hand-rolled lineage. 2. **Centralize.** Send events to Marquez or a lineage platform so the whole graph is queryable in one place. 3. **Validate coverage** — confirm datasets, edges, and column-level lineage render for critical flows. 4. **Impact analysis** — before changing/deprecating a table, traver