Problem: Sidekiq queues are not optimized for high volume federation

[jorgesat]

Sidekiq queues usually get stuck due to the large amount of requests when federating. To scale up the server it is often necessary to split the different queues, as proposed here.

Proposed solution:
To split sideqik into different processes for default, push, pull, mailers, scheduler.
Additionally, in more recent versions of mastodon an ingress queue has been implemented to handle toots from other instances. Perhaps that might help to optimize the traffic.

[weex]

I'm curious about how the queues get stuck but have also run across this splitting idea, probably from the same post.

[jorgesat]

In our case, the problematic one is pull that was getting stuck (once or twice a day if there was high activity).

At the moment, I have improved it by splitting the queues into:

• scheduler (with 2 threads)
• mailers (with 2 threads)
• push (with 8 threads)
• default (with 12 threads).
• pull x2 (with 65 threads each)

Although performing better, this is still not optimal, it gets stuck from time to time.