I read this in celery documentation :

Task.rate_limit http://celery.readthedocs.org/en/latest/userguide/tasks.html#Task.rate_limit

Note that this is a per worker instance rate limit, and not a global rate limit. To enforce a global rate limit (e.g. for an API with a maximum number of requests per second), you must restrict to a given queue.

How to put rate limit on celery queue?

Thanks for not down voting the question.

  • 1
    Celery docs were very vague on this :-( – Jack Tuck Sep 3 '15 at 12:45

You can set this limit in the flower > worker pane. there is a specified blank space for entering your limit there. The format that is suggested to be used is also like the below:

The rate limits can be specified in seconds, minutes or hours by appending “/s”, >“/m” or “/h” to the value. Tasks will be evenly distributed over the specified >time frame.

Example: “100/m” (hundred tasks a minute). This will enforce a minimum delay of >600ms between starting two tasks on the same worker instance.

  • 1
    You can set rate-limit on only task level on Flower. Celery does not support queue level rate-limiting. – Seunghoon Oct 30 '17 at 5:37

hey I am trying to find a way to do rate limit on queue, and I find out Celery can't do that, however Celery can control the rate per tasks, see this:

http://docs.celeryproject.org/en/latest/userguide/workers.html#rate-limits

so for a workaround, maybe you can set up one tasks per queue(which makes sense in a lot of situations), and put the limit on task.

Turns out it cant be done at queue level for multiple workers. IT can be done at queue level for 1 worker. Or at queue level for each worker.

So if u say 10 jobs/ minute on 5 workers. Your workers will process upto 50 jobs per minute collectively.

So to have only 10 jobs running at a time you either chose one worker. Or chose 5 workers with a limit of 2/minute.

  • 2
    How about some specifics - e.g. how exactly do you tell a worker to do only "2/minute"? – spookylukey May 19 '15 at 15:14
  • @spookylukey see my answer below – c-a Mar 7 '17 at 15:50
  • After searching for the answer of this question I found that rate-limiting is possible only on task level. Celery does not support queue level rate-limiting. – Seunghoon Oct 30 '17 at 5:33
  • so is there a way to get the number of workers dynamically and then a dynamic rate limit? – PirateApp Apr 3 at 4:30

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.