Skip to content

Django Tasks — Celery and Asynchronous Tasks

tasks.py is the transport layer for background operations. A task contains no business logic. It only queues the work and delegates execution to a Service.


Section What's inside
Why Celery The problem of blocking operations
Architecture Broker, Worker, Backend
Anatomy of a Task Basic structure, rules
Task = Transport Layer Why logic does not belong in a task
on_commit A guarantee of consistency with the DB
Retry and errors autoretry_for, max_retries, retry_backoff
Idempotency Tasks that are safe to run again
Common patterns Chains, Groups, Chords
Configuration Configuring Celery in Django
Anti-patterns What not to do and why
Template A copy-paste template for a new task

Why Celery

The problem of blocking operations

An HTTP request should complete within ~200–500 ms. Some operations take much longer:

Operation Time Problem
Sending an email 1–3 s The client waits
Processing a DEM file 30–120 s Nginx/client timeout
ICESat-2 request to SlideRule 10–60 s The worker is blocked
ML inference 5–30 s The client waits
Generating a PDF report 3–10 s The client waits

The solution: an asynchronous queue

HTTP Request
    │
    ▼
View → queues the task → immediately returns 202 Accepted
    │
    ▼ (independently, in the background)
Celery Worker → runs the task → stores the result

The client does not wait. The worker does not block HTTP requests.


Celery + Django architecture

Django App
    │
    │  task.delay(args)   ← queues the task
    ▼
Redis (Broker)            ← message queue
    │
    │  receives the task
    ▼
Celery Worker             ← a separate process, runs the task
    │
    │  result (optional)
    ▼
Redis / PostgreSQL (Backend)   ← stores the result if needed
Component Role
Broker Message queue (usually Redis or RabbitMQ)
Worker A separate Python process that runs tasks
Backend Result storage (Redis, PostgreSQL) — optional
Beat Scheduler for periodic tasks (similar to cron)

Anatomy of a Task

Basic structure

# apps/flood/tasks.py

from celery import shared_task


@shared_task
def notify_emergency_services_task(event_id: int) -> None:
    from apps.flood.selectors import flood_event_get_by_id
    from apps.flood.services import flood_event_notify

    event = flood_event_get_by_id(event_id=event_id)
    flood_event_notify(event=event)

Why the imports are inside the function

Module-level imports can cause circular dependencies when the Celery worker starts. Local imports inside the task function are a safe pattern.

# DANGEROUS — circular import when the worker starts
from apps.flood.services import flood_event_notify

@shared_task
def notify_task(event_id):
    ...

# SAFE — the import happens only when the task runs
@shared_task
def notify_task(event_id):
    from apps.flood.services import flood_event_notify
    ...

Passing arguments

# GOOD — pass simple types: int, str, list, dict
@shared_task
def process_flood_task(event_id: int) -> None:
    ...

# Call
process_flood_task.delay(event_id=42)


# BAD — passing Python objects (they do not serialize reliably)
@shared_task
def process_flood_task(event: FloodEvent) -> None:   # ❌
    ...

Celery serializes arguments via JSON. Pass IDs, strings, numbers — not ORM objects.


Task = Transport Layer

A task is a queue for delivering a call, not a place for logic.

Anti-pattern: logic in the task

# BAD — all the logic in the task
@shared_task
def process_flood_task(event_id: int):
    event = FloodEvent.objects.get(id=event_id)
    event.status = 'processing'
    event.save()

    # ML pipeline
    result = run_ml_model(event.geometry)
    event.risk_level = result['risk']
    event.status = 'done'
    event.save()

    # Email
    send_mail(
        subject='Flood processed',
        message=f'Event {event_id} done',
        from_email='system@example.com',
        recipient_list=[event.created_by.email],
    )

Problems: - The logic cannot be called from a View or the CLI without Celery - It cannot be tested without starting a worker - On retry, the whole pipeline runs again

Correct: the task delegates to a service

# GOOD — the task only transports the call
@shared_task
def process_flood_task(event_id: int) -> None:
    from apps.flood.selectors import flood_event_get_by_id
    from apps.flood.services import flood_event_process

    event = flood_event_get_by_id(event_id=event_id)
    flood_event_process(event=event)   # all the logic is in the service
# services.py — the logic is here, independent of Celery
@transaction.atomic
def flood_event_process(*, event: FloodEvent) -> FloodEvent:
    event.status = 'processing'
    event.save(update_fields=['status'])

    result = run_ml_model(event.geometry)
    event.risk_level = result['risk']
    event.status = 'done'
    event.save(update_fields=['risk_level', 'status'])

    transaction.on_commit(
        lambda: send_flood_report_task.delay(event.id)
    )

    return event

The same flood_event_process can be called from a View, a management command, a test — without Celery.


on_commit — Safe task launch

The problem without on_commit

@transaction.atomic
def flood_event_create(*, region_name: str, ...) -> FloodEvent:
    event = FloodEvent(...)
    event.save()

    # ❌ DANGEROUS:
    # The Celery worker may start BEFORE the transaction has closed.
    # The worker runs a SELECT — the record has not appeared in the DB yet.
    # The task fails with DoesNotExist.
    process_flood_task.delay(event.id)

    return event

The solution

@transaction.atomic
def flood_event_create(*, region_name: str, ...) -> FloodEvent:
    event = FloodEvent(...)
    event.save()

    # ✅ SAFE:
    # The task is queued ONLY after a successful commit.
    # If the transaction is rolled back, the task will not start at all.
    transaction.on_commit(
        lambda: process_flood_task.delay(event.id)
    )

    return event

Rule: any task.delay() inside @transaction.atomic must always go through on_commit.


Retry and error handling

autoretry_for — automatic retry on exceptions

import httpx
from celery import shared_task


@shared_task(
    autoretry_for=(httpx.HTTPError, ConnectionError),  # which errors to retry on
    max_retries=3,                                      # maximum number of attempts
    retry_backoff=60,                                   # exponential delay: base of 60 s
    retry_jitter=False,                                 # no random spread of the delay
)
def fetch_sliderule_data_task(job_id: int) -> None:
    from apps.satellite.services import icesat2_job_fetch

    icesat2_job_fetch(job_id=job_id)

With retry_backoff=60 and retry_jitter=False the delays will be: 60s → 120s → 240s. (retry_backoff=True means a base of 1 s: 1s → 2s → 4s, and countdown is an argument of apply_async()/self.retry(), not an option of the @shared_task decorator.)

Manual retry

@shared_task(bind=True, max_retries=5)
def process_dem_task(self, dem_id: int) -> None:
    from apps.dem.services import dem_process

    try:
        dem_process(dem_id=dem_id)
    except TemporaryProcessingError as exc:
        # Manual retry with a custom delay
        raise self.retry(exc=exc, countdown=30 * (self.request.retries + 1))

bind=True gives access to self — the current task instance with its metadata.

Handling after all attempts

from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)


@shared_task(
    autoretry_for=(Exception,),
    max_retries=3,
    default_retry_delay=60,
)
def critical_processing_task(job_id: int) -> None:
    from apps.flood.services import flood_job_process, flood_job_mark_failed

    try:
        flood_job_process(job_id=job_id)
    except Exception as exc:
        logger.error(f"Task failed for job_id={job_id}: {exc}")
        flood_job_mark_failed(job_id=job_id)
        raise   # lets Celery know the task failed → retry or FAILURE

Idempotency

A task may run several times (a network error, a retry, a duplicate message). Every task must be idempotent — running it again with the same arguments must not break the system.

Non-idempotent (dangerous)

@shared_task
def send_welcome_email_task(user_id: int) -> None:
    user = User.objects.get(id=user_id)
    send_mail(subject='Ласкаво просимо!', ...)
    # On retry — the email will be sent a second time

Idempotent (safe)

@shared_task
def send_welcome_email_task(user_id: int) -> None:
    user = User.objects.get(id=user_id)

    if user.welcome_email_sent:   # check before sending
        return

    send_mail(subject='Ласкаво просимо!', ...)

    User.objects.filter(id=user_id).update(welcome_email_sent=True)

Common patterns

A simple delay — fire and forget

# Run the task in the background
process_flood_task.delay(event_id=42)

# With keyword arguments
process_flood_task.apply_async(kwargs={'event_id': 42})

apply_async — extended options

from datetime import timedelta
from django.utils import timezone

# Delay execution by 10 minutes
process_flood_task.apply_async(
    kwargs={'event_id': 42},
    countdown=600,
)

# Run at a specific time
process_flood_task.apply_async(
    kwargs={'event_id': 42},
    eta=timezone.now() + timedelta(hours=1),
)

# Put it in a specific queue
process_flood_task.apply_async(
    kwargs={'event_id': 42},
    queue='high_priority',
)

Chain — sequential execution

from celery import chain

# Task 2 starts only after Task 1 completes successfully
pipeline = chain(
    fetch_sliderule_data_task.s(job_id=42),
    process_icesat2_points_task.s(job_id=42),
    finalize_icesat2_job_task.s(job_id=42),
)
pipeline.delay()

Group — parallel execution

from celery import group

# All tasks start in parallel
parallel_jobs = group(
    process_dem_task.s(dem_id=1),
    process_dem_task.s(dem_id=2),
    process_dem_task.s(dem_id=3),
)
parallel_jobs.delay()

Periodic Tasks (Celery Beat)

# config/celery.py

from celery.schedules import crontab

app.conf.beat_schedule = {
    'cleanup-old-jobs': {
        'task': 'apps.flood.tasks.cleanup_old_jobs_task',
        'schedule': crontab(hour=3, minute=0),   # every day at 3:00
    },
    'sync-satellite-data': {
        'task': 'apps.satellite.tasks.sync_icesat2_data_task',
        'schedule': crontab(minute='*/30'),       # every 30 minutes
    },
}

Configuration

config/celery.py

# config/celery.py

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.base')

app = Celery('project')
app.config_from_object('django.conf:settings', namespace='CELERY')

# Automatic discovery of tasks.py in all INSTALLED_APPS
app.autodiscover_tasks()

config/__init__.py

# config/__init__.py
from .celery import app as celery_app

__all__ = ('celery_app',)

settings.py

# settings.py

CELERY_BROKER_URL = 'redis://redis:6379/0'
CELERY_RESULT_BACKEND = 'redis://redis:6379/0'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'

# Important: always acknowledge the task AFTER execution
# (not after receiving it — protection against loss if the worker crashes)
CELERY_TASK_ACKS_LATE = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1

Starting the worker

# Basic start
celery -A config worker --loglevel=info

# With several queues and concurrency
celery -A config worker \
    --loglevel=info \
    --concurrency=4 \
    --queues=default,high_priority

# Celery Beat (periodic tasks)
celery -A config beat --loglevel=info

Anti-patterns

Business logic in a task

# BAD
@shared_task
def process_task(event_id):
    event = FloodEvent.objects.get(id=event_id)
    event.status = 'done'   # logic here
    event.save()

# GOOD
@shared_task
def process_task(event_id):
    from apps.flood.selectors import flood_event_get_by_id
    from apps.flood.services import flood_event_complete
    event = flood_event_get_by_id(event_id=event_id)
    flood_event_complete(event=event)

Passing an ORM object to a task

# BAD — an ORM object cannot be serialized to JSON (EncodeError), and with pickle it would carry stale state
process_task.delay(event=flood_event_object)

# GOOD — pass only the ID
process_task.delay(event_id=flood_event_object.id)

task.delay() without on_commit inside a transaction

# BAD — a race condition with the DB
@transaction.atomic
def create_event():
    event = FloodEvent.objects.create(...)
    process_task.delay(event.id)   # the worker may not find the record

# GOOD
@transaction.atomic
def create_event():
    event = FloodEvent.objects.create(...)
    transaction.on_commit(lambda: process_task.delay(event.id))

Calling a task synchronously in a View

# BAD — blocks the HTTP response
result = process_task(event_id=42)   # runs synchronously

# GOOD — queues it and returns immediately
process_task.delay(event_id=42)
return Response({'status': 'queued'}, status=202)

Table: what a task should and should not do

Action Should it?
Receive simple arguments (int, str) ✅
Run a SELECT to get an object ✅ (via a selector)
Delegate to a service ✅
Log errors ✅
Be idempotent ✅
Contain business logic ❌
Receive ORM objects as arguments ❌
Open @transaction.atomic ❌ (that is the service's role)
Return a Response ❌
Know about HTTP ❌

Template

# apps/<app>/tasks.py

from celery import shared_task
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)


@shared_task(
    autoretry_for=(Exception,),
    max_retries=3,
    retry_backoff=60,
    retry_jitter=False,
)
def <model>_<action>_task(<model>_id: int) -> None:
    from apps.<app>.selectors import <model>_get_by_id
    from apps.<app>.services import <model>_<action>

    logger.info(f"Starting <action> for <model>_id={<model>_id}")

    obj = <model>_get_by_id(<model>_id=<model>_id)
    <model>_<action>(obj=obj)

    logger.info(f"Completed <action> for <model>_id={<model>_id}")
# Calling it from a service via on_commit
@transaction.atomic
def <model>_create(*, <field>: <type>) -> <Model>:
    obj = <Model>(<field>=<field>)
    obj.full_clean()
    obj.save()

    transaction.on_commit(
        lambda: <model>_<action>_task.delay(<model>_id=obj.id)
    )

    return obj

Connection to the rest of the documentation

Topic File
Services (on_commit, the business logic a task delegates to) DJANGO_SERVICES.md
Selectors (reading data inside a task) DJANGO_SELECTORS.md
The full picture of the business layer DJANGO_SERVICES_SELECTORS.md
Project structure (infrastructure/celery/) DJANGO_PROJECT_STRUCTURE.md