← ClaudeAtlas

add-sourcelisted

Add a new data source (dlt pipeline, BigQuery native transfer, or on-prem database via Tailscale) to an existing Modern Data Stack. Invoke when the user wants to integrate a new SaaS, database, or Google service into the warehouse.
pol-cc/agentic-data-engineer · ★ 1 · AI & Automation · score 72
Install: claude install-skill pol-cc/agentic-data-engineer
# add-source > **Status**: v0.10.0 — dlt default; Airbyte/Singer documented escape. ## What this skill does Extends an existing MDS deployment with a new data source. The **default ingestion tool is dlt** (data load tool) — a Python library the agent drives directly. Three lanes: | Lane | When | How | |---|---|---| | **dlt pipeline** (default) | A SaaS API, or a cloud-hosted DB. Anything with an HTTP/REST API or a SQLAlchemy-reachable database. | Write a dlt pipeline script, `python load.py`, reconcile. See [`references/dlt-rest-api-source.md`](references/dlt-rest-api-source.md). | | **BigQuery native transfer** | The source is a Google service with first-party BQ export (GA4, Google Ads, Search Console) | `bq` CLI / BigQuery Data Transfer Service — never dlt, never Airbyte. See [`references/bq-native-transfer.md`](references/bq-native-transfer.md). | | **On-prem database via Tailscale** | A database physically inside the client's premises (SQL Server, MySQL, Postgres) | dlt `sql_database` source over the tailnet. See [`references/dlt-sql-database-source.md`](references/dlt-sql-database-source.md). | **Escape hatch (not the default):** when a SaaS API is too gnarly for a dlt `rest_api` config — pathological pagination, undocumented auth, a connector someone else already maintains better — fall back to a **maintained connector** (Airbyte standalone or a Singer tap) for *that one source*. Per source, not as a platform. See [`references/airbyte-api-gotchas.md`](references/a