I am using celery on rabbitmq. I have been sending thousands of messages to the queue and they are being processed successfully and everything is working just fine. However, the number of messages in several rabbitmq queues are growing quite large (hundreds of thousands of items in the queue). The queues are named celeryev.[...]
(see screenshot below). Is this appropriate behavior? What is the purpose of these queues and shouldn't they be regularly purged? Is there a way to purge them more regularly, I think they are taking up quite a bit of disk space.
5 Answers
You can use the CELERY_EVENT_QUEUE_TTL
celery option (only working with amqp), that will set the message expiry time, after which it will be deleted from the queue.
-
This link is now a dead link. Do you have an updated one? Is it now docs.celeryproject.org/en/latest/userguide/… ? Oct 28, 2019 at 14:40
For anyone else who is running into problems with a celeryev
queue becoming very large and threatening the disk space on your rabbitmq server, beware the accepted answer! Here's my suggestion. Just issue this command on your rabbitmq instance:
rabbitmqctl set_policy limit_celeryev_queues "^celeryev\." '{"max-length":1000000}' --apply-to queues
This will limit any queue beginning with "celeryev" to 1 Million entries. I did some experimenting with a stuck flower
instance causing a runaway celeryev queue, and setting CELERY_EVENT_QUEUE_TTL / CELERY_EVENT_QUEUE_EXPIRES did not help control the queue size.
In my testing, I started a flower
process, then SIGSTOP'ed it, and watched its celeryev queue start running away. Neither of these two settings helped at all. I confirmed SIGCONT'ing the flower
process would bring the queue back to 0 rapidly. I am not certain why these two knobs didn't help, but it may have something to do with how RabbitMQ implements these two settings.
First, the Per-Message TTL corresponding to CELERY_EVENT_QUEUE_TTL
only establishes an expiration time on each queue entry -- AIUI it will not automatically delete the message out of the queue to save space upon expiration. Second, the Queue TTL corresponding to CELERY_EVENT_QUEUE_EXPIRES
says that it "... guarantees that the queue will be deleted, if unused for at least the expiration period". However, I believe that their definition of "unused" may be too strict to kick in for e.g. an overburdened, stuck, or killed flower process.
EDIT: Unfortunately, one problem with this suggestion is that the set_policy ... apply-to queues
will only impact existing queues, and flower can and will create new queues which may overflow.
Celery use celeryev
prefixed queues (and exchange) for monitoring, you can configure it as you want or disable at all (celery control disable_events
).
-
Is there a way to reduce or limit event storage rather than turn it off altogether? Jul 22, 2013 at 5:54
-
Have you tried --maxrate and --frequency arguments (or some other?) which may be placed in CELERYEV_OPTS, I guess– pinepainJul 22, 2013 at 7:06
-
Were you able to make use of --maxrate or --frequency options? @pinepain Sep 2, 2020 at 17:15
You just have to set a config to your Celery.
If you want to avoid Celery from creating celeryev.*
queues:
CELERY_SEND_EVENTS = False # Will not create celeryev.* queues
If you need these queues for monitoring purpose (CeleryFlower for instance), you may regularly purge them:
CELERY_EVENT_QUEUE_EXPIRES = 60 # Will delete all celeryev. queues without consumers after 1 minute.
The solution came from here: https://www.cloudamqp.com/docs/celery.html
You can limit the queue size in RabbitMQ with x-max-length queue declaration argument