make all notifications async

[tarekziade]

Turns out we don't really have a use case for synchronous notifications: all the code that's triggered via events runs in isolation from the request/response flow and its result does not impact what we send back to the user.

For that reason we want to run all notifications asynchronously.

The simplest way to do it without an external service is to use a multiprocess.Pool where notifications jobs are added, and let workers pick the work to do.

The workers can be attached to the pyramid app as an utility for example.

Here's an implementation example of such a pool : https://github.com/AcrDijon/henet/blob/master/henet/workers.py

[tarekziade]

Implementation :

• add a workers.py module into cliquet, with a Workers class
• initialize it in cliquet.initialization so the registry gets a "workers" new attribute with n processes launched
• add a listener in cliquet.listeners that dumps event payloads into the workers pool and run asynchronously a function that runs in a separate process.

About the function that's executed asynchronously - I am not 100% sure about what should be the proper design. we could have a simple configuration system where a fully qualified name is declared, and gets the payload.

Config example:

kinto.event_listeners = background
kinto.event_listeners.background.processes = 4
kinto.event_listeners.background.functions = 
    myproject.mymodule.send_email
    myproject.mymodule.do_something_else

with send_email(event_payload)

[leplatrem]

It sounds good!

Regarding the config example, with what you propose it seems complicated to give extra parameters to functions or filter events by resource/actions.

An alternative is to move every listeners to this async implementation. The configuration of listeners does not change, expect that they are now always async.
If, for some reason, we need to run code synchronuously then we can still implement something that subscribes to the low-level Pyramid event and run code.

Configuring the worker class and number of processes now becomes another parameter along existing ones (use, actions, resources) and can have default (memory and # of CPU)

kinto.event_listeners = redis send_email
kinto.event_listeners.redis.use = cliquet.listeners.redis
kinto.event_listeners.redis.workers = cliquet.workers.memory
kinto.event_listeners.redis.processes = 1

kinto.event_listeners.send_email.use = myproject.mymodule.send_email
kinto.event_listeners.send_email.workers = cliquet_celery.workers
kinto.event_listeners.send_email.processes = 4
kinto.event_listeners.send_email.resources = bucket collection
kinto.event_listeners.send_email.actions = create delete
kinto.event_listeners.send_email.from = jeanlouis94@voila.fr

[tarekziade]

👍

[tarekziade]

There's one issue with your configuration proposal though: the pool of workers is initialized on startup independently from the listeners we've set up. Processes are also shared among listeners.

So "processes" should be a separate, global option. e.g.

Proposal:

kinto.background.processes = 4

kinto.event_listeners = redis send_email
kinto.event_listeners.redis.use = cliquet.listeners.redis
kinto.event_listeners.redis.workers = cliquet.workers.memory

kinto.event_listeners.send_email.use = myproject.mymodule.send_email
kinto.event_listeners.send_email.workers = cliquet_celery.workers
kinto.event_listeners.send_email.resources = bucket collection
kinto.event_listeners.send_email.actions = create delete
kinto.event_listeners.send_email.from = jeanlouis94@voila.fr

[tarekziade]

Implementing the feature in mozilla-services/cliquet#647

[leplatrem]

Archiving old issues