BC-269 Configure celery pool type and concurrency number.#201
BC-269 Configure celery pool type and concurrency number.#201olegtropinin wants to merge 1 commit intomasterfrom
Conversation
Makefile
Outdated
| .PHONY: run-celery | ||
| run-celery: | ||
| poetry run celery -A node.config.celery worker --loglevel=INFO | ||
| poetry run celery -A node.config.celery worker --loglevel=INFO --pool=eventlet --concurrency=30 |
There was a problem hiding this comment.
@olegtropinin let's go with prefork or process and autoscale for now, we will optimize with async approach later if needed
There was a problem hiding this comment.
actually, these comments apply to docker-compose.yml only. here we need just keep it as it was, because it is OK for development
There was a problem hiding this comment.
I found information that "eventlet" pool type have greater concurrency than prefork. Also it's good for API calls. It was a case why I selected prefork.
I removed "eventlet" as you said. It has "prefork" default pool now.
You are right no need to have too many threads for development. I removed concurrency argument from Makefile (make run-celery).
docker-compose.yml
Outdated
| TNB_CELERY_BROKER_URL: 'amqp://guest:guest@celery-broker:5672//' | ||
| env_file: .env | ||
| command: poetry run celery -A node.config.celery worker --loglevel=INFO | ||
| command: poetry run celery -A node.config.celery worker --loglevel=INFO --pool=eventlet --concurrency=30 |
There was a problem hiding this comment.
let's have concurrency depend on max PV schedule length (read from env var that corresponds to django setting)
There was a problem hiding this comment.
Concurrency value is now same as TNB_NODE_SCHEDULE_CAPACITY environment value.
olegtropinin
force-pushed
the
BC-269-increase-celery-tasks
branch
from
dbb6bb7 to
9660f45
Compare
2 participants
No description provided.