etl-incremental-pipeline-builderlisted

# ETL Incremental Pipeline Builder Build and run idempotent, incremental Extract-Transform-Load pipelines entirely on the Python standard library, so the agent can exercise the full lifecycle - schema, load, delta, verify - inside a sandbox without any external app. The reference implementation is a complete `sqlite3 + csv + json + argparse` module you can run verbatim. ## 1. System Architecture & Prerequisites - **Runtime**: CPython 3.9+ (`sqlite3`, `csv`, `json`, `argparse`, `tempfile`, `hashlib`, `os`, `sys`). No third-party packages, no database server, no scheduler, no MCP. - **Components**: - `staging` layer: raw CSV snapshots read once per run. - `control` layer: `_etlctl` table holding the high-watermark (`last_watermark`) per source. - `warehouse` layer: idempotent `main` offsets with `batch_id`, natural key + `etl_updated_at`. - **Increment model**: delta = rows whose `src_updated` watermark is strictly greater than the stored watermark. Re-running the same snapshot is a no-op (idempotent). - **Idempotency**: a `batch_id = sha256(source|max_watermark|run_epoch)` is written; a repeated run with the same batch_id returns 0 changed rows. - **Schema evolution**: columns found in the CSV but missing in the warehouse are added via `ALTER TABLE ... ADD COLUMN` before insert. ## 2. Input/Output Data Contracts | Artifact | Location | Format | | --- | --- | --- | | Source snapshot (input) | `data/<source>.csv` or `--snapshot path.csv` | CSV UTF-8, header row, `reco