marp	theme
true	gaia

Spokane Python User Group

Background Task Processing:

Celery vs. Dramatiq

Joseph Riddle

---

Raffle

Feedback for free JetBrains

https://forms.office.com/r/EFnAfpu3Ue

---

Outline

• Definitions
• What is Celery
• What is Dramatiq
• Celery vs. Dramatiq
• Demo

---

Message

A unit of communcation between computer processes.

Task

An asynchronous unit of work.

---

Message Queue

Provides fast asynchronous communication between software components.

Temporarily stores messages, with endpoints for software services to connect to the queue.

---

https://better.engineering/message-queues/

---

Message Broker

An architectural pattern for message validation, transformation, and routing.

Decoupled communication.

Popular message brokers include RabbitMQ, Apache Kafka, and more...

Cloud provider options include Amazon MSK, Azure Event Hubs, Google Pub/Sub, and more...

---

RabbitMQ

RabbitMQ is the most widely deployed open source message broker.

https://www.rabbitmq.com/

Supports AMQP and others

Official Docker image

---

AMQP

The Advanced Message Queuing Protocol (AMQP) is an open standard for passing business messages between applications or organizations.

AMQP Primer

ISO 19464

---

Redis

An open-source, networked, in-memory, key-value data store with optional durability.

Can be used as a message broker.

Official Docker image

---

Celery

Celery is an open source asynchronous task queue or job queue which is based on distributed message passing. While it supports scheduling, its focus is on operations in real time.

The de facto Python task queue library.

GitHub repo created in 2009

17.7K⭐

---

Dramatiq

Dramatiq is a background task processing library for Python with a focus on simplicity, reliability and performance. Created by a user of Celery.

Pronounced the same as "dramatic".

GitHub repo created in 2017

2.7K⭐

---

Use Cases

• IO bound tasks
  • Background tasks for web API
  • Sending emails
  • Connecting to other APIs
  • Long running tasks
• CPU/GPU bound tasks
  • Distributed machine learning batch job
  • Can distribute to multiple workers
• Scheduling tasks

---

Celery vs. Dramatiq

	Celery	Dramatiq
Broker Support	RabbitMQ Redis Amazon SQS more...	RabbitMQ Redis In-memory more...
Result Store Support	Redis Memcached MongoDB File system more...	Redis Memcached

---

Celery vs. Dramatiq

	Celery	Dramatiq
Scheduling	Celery Beat	Accomplished via APScheduler
Monitoring	Flower	Yes *
Chaining	Yes	Yes

https://dramatiq.io/motivation.html

---

Celery Code

from celery import Celery

app = Celery('tasks', broker='pyamqp://rabbitmq//')

@app.task
def count_to(n):
    for i in range(n):
        print(i)

---

Dramatiq Code

import dramatiq
from dramatiq.brokers.rabbitmq import RabbitmqBroker

rabbitmq_broker = RabbitmqBroker(url='amqp://guest:guest@rabbitmq')
dramatiq.set_broker(rabbitmq_broker)

@dramatiq.actor
def count_to(n):
    for i in range(n):
        print(i)

---

Best Practices

• Pass small amounts of data as messages. Prefer passing IDs instead of large objects, when possible.
• Don't use result backends if you don't have to.
• Make tasks idempotent

---

Other Python Message Queue projects

• RQ (Redis Queue) Simple + Redis
• Taskmaster Simple + One-off tasks
• Huey Lightweight + Redis
• Kuyruk Simple + RabbitMQ
• Django Carrot Simple + Django

See https://www.fullstackpython.com/task-queues.html

---

Questions?

---

Raffle

Feedback for free JetBrains

https://forms.office.com/r/EFnAfpu3Ue

---

References

More on Message Queues

Dramatiq with docker-compose

Celery with Kubernetes