sota-data-engineeringlisted

# SOTA Data Engineering Expert rules for analytical data systems: pipelines, streaming, warehousing, lakehouse storage, data quality, and operations. OLTP schema/query/index craft is owned by `sota-databases` — reference it, do not duplicate it. Outbox and event-driven service patterns live in `sota-architecture`; backpressure mechanics in `sota-async-concurrency`; PII handling in `sota-privacy-compliance` (or `sota-code-security` if absent). Two modes. Pick by intent, then load only the `rules/` files the task needs. ## BUILD mode Use when designing or implementing pipelines, models, streaming jobs, or storage layouts. 1. **Size the problem first.** Read `rules/01-architecture-and-modeling.md` before choosing tools. Most "big data" is small data; DuckDB/Polars on one node before a distributed engine. ELT into a warehouse/lakehouse is the default shape, not a decision to revisit per pipeline. 2. **Idempotency is the prime directive.** Every pipeline you write must be safe to rerun for any interval at any time. No blind appends. Design the write strategy (partition overwrite / merge key / insert-overwrite) before the transform logic (`rules/02-pipelines-and-orchestration.md`). 3. **Batch unless a consumer needs sub-minute data.** Justify streaming in writing before building it (`rules/03-streaming-and-cdc.md`). 4. **Quality checks ship with the pipeline, not after.** Every new model gets freshness, volume, uniqueness, and not-null checks tiered blo