designing-medallion-architecturelisted

# Designing Medallion Architecture ## When to use - Organizing a lakehouse (Databricks/Iceberg/Delta) into layered zones. - Deciding what transformation belongs in bronze vs silver vs gold. - Refactoring a flat, hard-to-debug pipeline into clear layers. - Do NOT use for dimensional modeling details (use `modeling-dimensional-data`). ## The three layers - **Bronze (raw)** — ingested data as-is, append-only, with load metadata (source, ingest time, file). Immutable; enables replay without re-pulling. - **Silver (cleaned/conformed)** — deduplicated, typed, validated, joined into conformed entities. The trustworthy, queryable base. - **Gold (business)** — aggregated marts, metrics, and dimensional models serving BI/ML. ## Workflow ``` - [ ] Land raw immutably in bronze with lineage metadata - [ ] Clean/dedupe/validate into silver; enforce schema + quality here - [ ] Model + aggregate into gold for consumers - [ ] Make each layer transition idempotent (overwrite/upsert by key) - [ ] Keep heavy business logic in gold, not bronze ``` 1. **Bronze = capture, not transform.** Store raw exactly as received so you can reprocess when logic changes. No business rules here. 2. **Silver = trust.** Deduplicate, cast types, apply data contracts and quality checks, and conform entities. Most `implementing-data-quality-checks` gates live here. 3. **Gold = serve.** Build dimensional models and aggregates (`modeling-dimensional-data`) for dashboards and features. 4. **Idem