loom-background-jobs

# Background Jobs ## Overview Reliable async task execution: enqueue work, process it in workers decoupled from the request cycle, and survive crashes/retries without corrupting state. This file covers job queues, retries/backoff, DLQs, scheduling, worker pools, and delivery guarantees. For pub/sub, event sourcing, CQRS, sagas, and streaming brokers (Kafka/Pulsar), see **loom-event-driven** — don't reimplement those here. ## The two invariants everything hangs off 1. **At-least-once is the default. Design every handler to be idempotent.** Redis-backed queues (Sidekiq, BullMQ, Celery+Redis), SQS standard, and any queue that retries on crash *will* deliver a job more than once. "Exactly-once delivery" does not exist over an unreliable network; the achievable goal is exactly-once *effect* = at-least-once delivery + idempotent handler. 2. **Ack after success, never before.** The job must stay owned by the worker until the side effect is durably committed. Ack-then-process = at-most-once = silent data loss on crash. Process-then-ack = at-least-once = duplicates you dedup away. Always choose the latter. ### Idempotency: the non-negotiable pattern Derive a stable key from the job's *business identity* (not a random UUID per enqueue), and gate the side effect on it. ```python def handle(job): key = job["idempotency_key"] # e.g. f"charge:{order_id}" if store.setnx(f"done:{key}", "1", ex=7*86400): # first time → claim do_side_effect(job) ...