1

I have question about CELERYD_CONCURRENCY and CELERYD_PREFETCH_MULTIPLIER
Because my english is not well to understand the official site description,
I want to make sure it

I set CELERYD_CONCURRENCY=40 I think it will use 40 workers to do things
But I usually see INFO/MainProcess ,seldom see INFO/Worker-n
Is it because the task is fast,so it didn't have to assign to worker??

Here is a task architecture :
I have a period_task is celery period_task , and mail_it is normal celery task

@shared_task
def period_task():
    do_something()
    ....
    for mail in mail_list:
        mail_it.delay(mail)

enter image description here

And the second question is CELERYD_PREFETCH_MULTIPLIER ,the default value is 4
Is it means that each worker can get 4 tasks from queue one time ??? So I have 40 worker,I can get 40*4 task????

1 Answer 1

3

My understanding:

CELERYD_CONCURRENCY: This is the number of THREADS/GREENTHREADS a given worker will have. Celery calls these "processes". This is the number of tasks a single worker can execute in parallel. I believe Celery creates this numbe PLUS ONE internally, and that the additional 1 is for actually managing/assigning to the others (in your case, 40 of them!). In my experience, you likely don't need/want 40 (closer to 1 or 2 per CPU), but your mileage may vary.

CELERYD_PREFETCH_MULTIPLIER: Prefetch is how many tasks are reserved per "process" according to the docs. It's a bit like a mini-queue just for that specific thread. This would indeed mean that your ONE started worker would potentially 'reserve' 40 * 4 tasks to do. Keep in mind that these reserved tasks cannot be "stolen" or sent to another worker or thread, so if they are long running you may wish to disable this feature to allow faster stations to pickup the slack of slower ones.

If this isn't clear in your current setup, I might suggest adding a sleep() to your task to be able to observe it.

3
  • Thank you very much. What does but your mileage may vary means?? Jan 18, 2016 at 6:08
  • 3
    Your Mileage May Vary (you might see it shortened to 'YMMV' on the internet) means your experience may be different than mine. I say this here because I use fewer threads and long-running tasks while you are using many threads and short tasks, and so my experience with Celery could be very different than yours. I simply don't have the -exact- experience here to tell you if the 40 threads is a good or bad practice for your needs. Sorry about using that phrase, as I realize now that it is a bit of a weird sentence/phrase if English is a second language. Jan 18, 2016 at 14:09
  • Thank you so much .I got it Jan 20, 2016 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.