data-pipeline-architect

# Data Pipeline Architect ## Overview This skill helps you design and review production-grade data pipelines. It covers the full lifecycle: ingestion, transformation, idempotency and backfills, schema evolution, orchestration, and data quality. It is opinionated toward modern ELT (load raw, transform in-warehouse) but supports classic ETL where it fits. Keywords: ETL, ELT, data pipeline, ingestion, idempotency, backfill, schema evolution, schema drift, CDC, incremental load, watermark, orchestration, Airflow, Dagster, Prefect, dbt, data quality, freshness, dedup, late-arriving data, partitioning, medallion, bronze silver gold. Use this skill to produce a concrete pipeline design (a design doc), to review an existing pipeline against best practices, or to generate skeleton DAGs/models and data-quality checks. ## Decision: ETL vs ELT Default to **ELT** when the destination is a modern columnar warehouse/lake (Snowflake, BigQuery, Redshift, Databricks, DuckDB). Land raw data first, transform with SQL/dbt. Use **ETL** when: the destination can't transform cheaply, you must mask/drop PII before it lands (compliance), or you transform in-flight for a stream. See `references/etl-vs-elt.md`. ## Workflow Follow these steps in order. Produce the design document in `templates/pipeline-design.md` as you go. 1. **Clarify requirements.** Capture: sources, destination, SLA/freshness (real-time, hourly, daily), volume (rows/day, GB/day), data sensitivity (PII?), and consumers (BI, M...