async-processing

# Async Processing Skill > **Expertise:** Task queues (Celery, ARQ, Dramatiq), Kafka/NATS consumers, background jobs, retry strategies, idempotency, dead-letter queues. ## When to Use Async Processing ``` Use async when: ✅ Operation takes > 200ms (send email, resize image, call slow 3rd party) ✅ Work can be retried independently (payment webhook, notification) ✅ Decoupling producers from consumers is required ✅ Fan-out to multiple consumers needed Keep synchronous: ❌ Response depends on the result (user sees outcome immediately) ❌ Must be transactional with the triggering DB write ``` ## Task Queue (Celery + Redis) ```python # tasks.py from celery import Celery from celery.utils.log import get_task_logger app = Celery("myapp", broker="redis://localhost:6379/1", backend="redis://localhost:6379/2") app.conf.update( task_serializer="json", result_expires=3600, task_acks_late=True, # Ack after completion, not on receive task_reject_on_worker_lost=True, task_default_retry_delay=60, # 1 min base delay task_max_retries=5, ) logger = get_task_logger(__name__) @app.task(bind=True, max_retries=5, default_retry_delay=30) def send_order_confirmation(self, order_id: int) -> None: try: order = Order.objects.get(id=order_id) email_service.send_confirmation(order) logger.info("email.sent", extra={"order_id": order_id}) except EmailServiceError as exc: # Exponential backoff: 30s, 60s, 120s, 240s, 480s d...