data-pipeline-builder

# Data Pipeline Builder Design and implement production-grade ETL/ELT data pipelines: take data sources, a destination, and transformation requirements, then produce a complete pipeline specification plus all implementation files needed to run it. ## Contents - `references/project-structure.md` -- output file layout, architecture pattern selection, component selection. - `references/python-patterns.md` -- Python code standards and base extractor/transformer/loader/retry patterns. - `references/quality-checks.md` -- composable data quality check framework and built-in checks. - `references/orchestration-config.md` -- Airflow DAG, pipeline config YAML, and monitoring/alerting patterns. - `references/spec-template.md` -- the `data-pipeline-spec.md` output template. ## Workflow 1. Gather requirements. If the user gave clear requirements, proceed to design. Otherwise ask targeted questions: data sources (databases, APIs, files, streams); destination (warehouse, lake, database); transformations (joins, aggregations, filters, business rules); freshness requirement (real-time, hourly, daily); technology preferences (Airflow, dbt, Spark, cloud provider); data quality and compliance requirements. 2. Analyze and design. Catalog each source (connection type, auth, schema, volume, CDC availability, rate limits). Define the destination (platform, schema design, partitioning, clustering, access patterns). Map transformations (field mappings, business logic, type conversions, joins, ag...