data-engineering

# Data engineering Pipelines are production systems whose failures are quiet. A broken service pages someone; a broken pipeline produces plausible numbers that people act on for a week. This is movement and transformation. Schema and semantics belong to `data-analytics:data-modeling`, policy and stewardship to `data-analytics:data-governance`. ## Land raw, transform downstream Keep an immutable copy of source data exactly as received. Transformation logic will be wrong at some point, and raw data is what lets you reprocess rather than re-request from a source that may no longer have it. Business logic belongs downstream where it is visible and testable, not buried in ingestion. The exception is transformation required for privacy — minimization, pseudonymization, dropping fields you have no basis to hold — which belongs at ingest precisely because raw storage is what the obligation attaches to. See `legal-risk:privacy-and-data-protection`. ## Idempotence is the property that matters Every pipeline will be re-run: after a failure, after a fix, after a late-arriving correction. A re-run that double-counts is worse than a failure, because it produces a wrong answer silently. Design for exactly-once effect at the destination — deterministic keys, merges rather than blind appends, partitioned overwrites. Then re-running is safe and recovery stops being frightening. ## Late, duplicate and out-of-order data Real sources deliver all three. Decide explicitly, per pipeline: h...