Task Queues

Taskiq's docs call it an asyncio Celery implementation: you mark tasks with @broker.task and send them with await task.kiq(). Since sending is async only, its docs point fully synchronous projects to Celery or Dramatiq. For production, its docs highly recommend taskiq-aio-pika or taskiq-nats as the broker and taskiq-redis as the result backend. In a FastAPI app, taskiq-fastapi connects the broker to your app.

Celery is a task queue with batteries included: celery beat runs periodic tasks, and Django support comes out of the box. It runs on RabbitMQ, Redis, or Amazon SQS. Its docs call RabbitMQ an excellent choice for production and warn that Redis is more likely to lose data on a sudden shutdown or power failure. Its protocol has clients in other languages, so non-Python code can send tasks too. Celery doesn't support Windows, though.

RQ is a simple job queue on Redis or Valkey, built for a low barrier to entry. Any Python function call can go on a queue, and there are no queues, exchanges, or routing rules to set up first. Jobs are pickled, so RQ is Python-only. Put job functions in a module the worker can import, not in __main__, and run workers on the same source code as your app. Start workers with rq worker --with-scheduler to run jobs you schedule with enqueue_in or enqueue_at.

Huey is a lightweight alternative with zero dependencies, and its queue can live in Redis, Postgres, SQLite, files, or memory. Its docs match the storage to your workload: Redis for busy workloads, SQLite for moderate ones without a separate server, and Postgres when your app already runs it. Recurring tasks come built in with the periodic_task() decorator. For Django, Huey has its own integration, and it provides a production backend for Django's task framework.

Dramatiq aims for sane defaults for most SaaS workloads on RabbitMQ or Redis. Mark a function with @dramatiq.actor and enqueue it with .send(), passing only JSON-encodable arguments. When an actor raises, Dramatiq retries it with exponential backoff. It acknowledges a message only after processing it, while Celery by default does so just before running the task. For cron-style jobs, pair it with a separate scheduler. Dramatiq is licensed under the LGPL.

Make tasks safe to run twice: after a worker failure, the same message can arrive again. Pass a task the ID of a database row rather than the object, and re-fetch it when the task runs, since old data leads to race conditions. In Django, enqueue a task only after the transaction commits, with Celery's delay_on_commit() or Huey's on_commit_task(). With celery beat, Huey, or Taskiq, run only one scheduler for periodic tasks, or you'll get duplicate tasks. In production, run workers under a process manager.