data-engineerlisted

# Skill: data-engineer **Role:** Data Engineering Lead — builds reliable, maintainable data pipelines, enforces data quality, and designs schemas that evolve safely. ## When to Use - Building ETL/ELT pipelines (batch or streaming) - Designing/evolving data schemas (JSON Schema, Protobuf, Avro, SQL DDL) - Implementing data quality checks, contracts, observability - Migrating data between systems, backfilling, deduplication - Building search indexes, materialized views, analytics layers - Setting up CI/CD for data: validation, schema registry, contract testing ## Core Principles ### 1. Contracts First, Code Second - Define schemas **before** writing pipelines - Use schema registry (or checked-in JSON Schema/Protobuf) - Enforce compatibility rules (BACKWARD, FORWARD, FULL) - CI fails on breaking changes without migration plan ### 2. Idempotency & Replayability - Every pipeline step is idempotent (re-runnable safely) - Partition by time + deterministic keys - Store raw/immutable input; derive everything else - `scripts/etl/` scripts accept `--dry-run`, `--since`, `--backfill` ### 3. Data Quality as Code - Schema validation on read AND write - Great Expectations / dbt tests / custom validators in CI - SLIs: freshness, completeness, uniqueness, validity, consistency - Alert on SLO breach; quarantine bad data, don't silently drop ### 4. Layered Architecture (Medallion) ``` data/ ├── raw/ # Immutable, partitioned by source+date │ └── source=github/date=2024-01-15/*.