orchestrating-prefect-flowslisted
Install: claude install-skill Unknown-333/awesome-data-engineering-skills
# Orchestrating Prefect Flows
## When to use
- Writing or refactoring Prefect flows and tasks.
- Configuring retries, caching, parameters, concurrency, or deployments/schedules.
- Migrating standalone Python scripts into managed orchestration.
- Do NOT use for Airflow (use the Airflow skills) or Dagster assets.
## Workflow
```
- [ ] Wrap the pipeline in a @flow; decompose steps into @task
- [ ] Parameterize by run window, not now(); keep tasks idempotent
- [ ] Add retries + retry_delay on flaky/external tasks
- [ ] Cache pure tasks by input to skip redundant work
- [ ] Create a deployment with a schedule; store secrets in blocks
```
1. **Flows and tasks.** A `@flow` is the orchestrated unit; `@task` functions are
the retryable, observable steps. Return values pass data between tasks.
2. **Idempotency + parameters.** Pass the processing window as a parameter and make
writes upsert/overwrite so retries and reruns are safe.
3. **Retries** on tasks that call networks/warehouses; transient failures self-heal.
4. **Caching** — cache deterministic tasks keyed on inputs to avoid recomputation.
5. **Deployments** attach a schedule and infrastructure; **blocks** hold
connections/secrets instead of hard-coding them.
## Patterns
**Flow with retries and idempotent load:**
```python
from prefect import flow, task
from datetime import timedelta
@task(retries=3, retry_delay_seconds=30)
def extract(run_date):
return fetch_orders(run_date) # window is a parameter