debugging-airflow-pipelineslisted

# Debugging Airflow Pipelines ## When to use - A task failed, is stuck in `queued`/`scheduled`, or became a zombie. - The scheduler isn't launching runs, or a DAG won't trigger. - XCom pull errors, dependency deadlocks, or pool/concurrency starvation. - Do NOT use for writing new DAGs (use `authoring-airflow-dags`). ## Workflow ``` - [ ] Read the task log first (Grid view -> task -> Logs) - [ ] Check task state and why: failed, up_for_retry, queued, or none - [ ] Localize: task-level bug vs scheduler/executor vs resource limit - [ ] Fix root cause, then clear the task to re-run ``` 1. **Read the task log.** The Grid/Graph view → failed task → Logs shows the real exception nearly every time. 2. **Check the state and reason.** `queued` for a long time is usually a resource/executor issue, not a code bug. 3. **Localize** using the table below. 4. **Re-run** by clearing the task instance (and downstream if needed) rather than re-triggering the whole DAG. ## Patterns **Task keeps failing** — read the log; fix the exception; confirm `retries` are set so transient errors self-heal. Use `on_failure_callback` for alerting. **Task stuck in `queued`/`scheduled`:** - Worker capacity exhausted, or a `pool` is full → check pool slots and `max_active_tasks`/parallelism. - Celery/Kubernetes executor not picking up → check worker health and the message broker/queue. **Zombie tasks** (process died, heartbeat lost) → often OOM or a killed worker. Check worker memory/logs;