designing-backfills-and-replayslisted

# Designing Backfills and Replays ## When to use - Loading history for a new model/pipeline. - Reprocessing past windows after fixing a transformation bug. - Replaying events or re-deriving a table from raw. - Do NOT use for normal incremental runs (use `building-dbt-models` / `authoring-airflow-dags`). ## Why it deserves care Backfills are the most dangerous routine operation in data engineering: they rewrite history. Done wrong, they double-count, change yesterday's numbers, or overload production. Done right, they are boring and repeatable. The prerequisite is idempotency (see `writing-idempotent-transformations`). ## Workflow ``` - [ ] Confirm the transform is idempotent (rerun == run once) BEFORE backfilling - [ ] Define the exact window and partition granularity - [ ] Dry-run one partition; verify counts/sums vs source - [ ] Run partition-by-partition (bounded parallelism), not all at once - [ ] Isolate backfill compute from production workloads - [ ] Verify totals and reconcile; then resume normal scheduling ``` 1. **Prove idempotency first.** If re-running a window can change results, fix that before touching history. Backfilling a non-idempotent job multiplies data. 2. **Scope precisely.** Exact start/end and the partition unit (day/hour/region). 3. **Dry-run one partition** and reconcile against source before scaling out. 4. **Chunk the run.** Process partitions in bounded batches so you can monitor, pause, and resume — never one giant unbounded job.