celery-patternslisted

# Celery Patterns Distributed task queue patterns for Python applications. ## When to Activate - Offloading long-running tasks (email, PDF generation, data processing) - Scheduling periodic tasks (cron-like) - Building data processing pipelines - Implementing retry logic for external API calls - Setting up task monitoring and alerting ## Setup ### Configuration ```python # config/celery.py from celery import Celery from app.config import settings celery_app = Celery( "worker", broker=settings.REDIS_URL, backend=settings.REDIS_URL, ) celery_app.conf.update( task_serializer="json", accept_content=["json"], result_serializer="json", timezone="UTC", enable_utc=True, task_track_started=True, task_time_limit=300, # Hard limit: 5 minutes task_soft_time_limit=240, # Soft limit: 4 minutes worker_prefetch_multiplier=1, # Disable prefetching for fair scheduling worker_max_tasks_per_child=1000, # Restart worker after 1000 tasks (memory leak protection) task_acks_late=True, # Acknowledge after execution (at-least-once) task_reject_on_worker_lost=True, result_expires=3600, # Results expire after 1 hour ) # Auto-discover tasks in all apps celery_app.autodiscover_tasks(["app.tasks"]) ``` ### Django Integration ```python # config/celery.py (Django) import os from celery import Celery os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.production") app = Celery("myproj