data-pipeline-etl-expert

# Data Pipeline & ETL Expert [English](#english) | [Bahasa Indonesia](#bahasa-indonesia) --- <a name="english"></a> ## English ### Description A specialized skill for building robust data architectures, Analytics Engineering, and ETL (Extract, Transform, Load) or ELT pipelines. It covers modern data stack orchestration (Airflow, Dagster), transformation tools (dbt), and high-performance OLAP databases (BigQuery, Snowflake, ClickHouse, DuckDB). ### Trigger Conditions - When designing reporting dashboards or analytics infrastructure for a SaaS. - When moving large volumes of data from transactional databases (PostgreSQL/MySQL) to a data warehouse. - When the user asks about "dbt", "Airflow", "ELT", or "Analytics Engineering". - When building local or edge analytics using DuckDB. ### Core Architectural Guidelines #### 1. ELT over ETL Prefer Extract-Load-Transform (ELT) over traditional ETL. - **Extract & Load**: Use tools like Airbyte or Fivetran to dump raw data directly into the Data Warehouse. - **Transform**: Perform transformations *inside* the Data Warehouse using SQL (via dbt) to leverage the warehouse's massive compute power. #### 2. Analytics Engineering with dbt Treat SQL like software engineering. - Use `dbt` (Data Build Tool) to version control your SQL transformations. - Implement tests (`not_null`, `unique`) on critical tables. - Use Jinja templating in dbt to DRY up complex SQL queries. #### 3. Data Orchestration (Airflow vs Dagster) - **Apache Airflow**:...