designing-data-contractslisted

Define and enforce data contracts between producers and consumers — explicit schema, semantics, ownership, SLAs, and versioning — to prevent silent upstream changes from breaking downstream pipelines. Use when a producer schema change could break consumers, defining an interface between teams/services and the warehouse, or adding schema enforcement at ingestion.
Unknown-333/awesome-data-engineering-skills · ★ 16 · Data & Documents · score 68

Install: claude install-skill Unknown-333/awesome-data-engineering-skills

# Designing Data Contracts ## When to use - An upstream (service, event, API, file) feeds downstream pipelines and a change could break them silently. - Defining the interface between a producing team/system and the warehouse. - Adding schema/quality enforcement at the ingestion boundary. - Do NOT use for internal model-to-model changes within one dbt project (use tests + `handling-schema-evolution`). ## What a contract specifies - **Schema**: fields, types, nullability, and allowed values. - **Semantics**: what each field means and its unit/grain. - **Guarantees**: freshness/SLA, volume expectations, uniqueness of keys. - **Ownership**: who produces it and who to contact. - **Versioning + change policy**: how breaking changes are communicated. ## Workflow ``` - [ ] Write the contract as a versioned, checked-in schema (not tribal knowledge) - [ ] Enforce it at the ingestion boundary (validate on arrival) - [ ] Classify changes: additive (safe) vs breaking (needs a new version) - [ ] On violation, reject/quarantine and alert the producer - [ ] Version and communicate breaking changes ahead of time ``` 1. **Make it explicit and versioned.** Store the contract as code (JSON Schema, Avro/Protobuf schema, or a YAML spec) next to the pipeline, reviewed like any API. 2. **Enforce at the boundary.** Validate incoming data against the contract on arrival; reject or quarantine violations instead of loading them. 3. **Classify changes.** Additive/optional fields = ba