processing-streaming-datalisted

# Processing Streaming Data ## When to use - Building or debugging Kafka / Spark Structured Streaming / Flink pipelines. - Choosing delivery semantics and making consumers idempotent. - Windowing on event time and handling late/out-of-order events. - Do NOT use for batch ELT (use `building-ingestion-pipelines`). ## Workflow ``` - [ ] Choose delivery semantics; make effects idempotent regardless - [ ] Use event time (not processing time) with watermarks - [ ] Configure checkpointing for recovery - [ ] Decide late-data policy (allowed lateness -> update, drop, or side output) - [ ] Size partitions/parallelism to the throughput ``` 1. **Delivery semantics.** At-least-once is the common default (duplicates possible on retry). Exactly-once needs transactional sinks/offsets. Either way, **make the downstream effect idempotent** (upsert by key) so duplicates don't corrupt state — this is more robust than relying on exactly-once alone. 2. **Event time + watermarks.** Window by when the event happened, not when it was processed; a watermark bounds how long to wait for stragglers. 3. **Checkpoint** so a failed job resumes from the last committed offset/state instead of reprocessing everything or losing data. 4. **Late data policy** — allowed lateness updates windows; beyond it, drop or route to a side output/dead-letter for reconciliation. ## Patterns **Idempotent consumer** — key the sink write on a stable event id so replays upsert rather than duplicate: ```