job-reliabilitylisted

# Job Reliability ## Purpose Provide a standardized methodology for making background jobs and message consumers reliable under the delivery guarantee that nearly every queue actually offers: **at-least-once**. That guarantee has two consequences every consumer must handle: **duplicates** (the same message is delivered more than once) and **partial failure** (the worker crashes after doing the work but before acknowledging). Code that ignores either will silently double-charge, double-send, lose messages, or stall. This skill catalogs the gaps and the fixes, statically, from the worker source. ## The at-least-once contract - **Duplicates are guaranteed, eventually.** A network blip, a redelivery after a visibility timeout, or a consumer restart re-delivers a message. → **Make consumers idempotent** (see `references/dedup-strategies.md`). - **Ack means "I'm done," not "I received it."** Ack/commit the offset **after** the side effect succeeds, never before. Ack-then-work loses the message on a crash. - **Poison messages exist.** A message that always fails must not retry forever, route it to a **dead-letter queue** after a bounded number of attempts. - **Retries need backoff + jitter.** Immediate infinite retries create a thundering herd against a failing dependency. ## The reliability checklist (audit every consumer against this) | Axis | The gap | The fix | |---|---|---| | Idempotency | Side effect runs again on redelivery (double charge/email) | Dedup store keyed by mes