3

I'm using Django==2.2.24 and celery[redis]==4.4.7.

I want to get the length of my celery queues, so that I can use this information for autoscaling purposes in AWS EC2.

I found the following piece of documentation:

https://docs.celeryproject.org/en/v4.4.7/userguide/monitoring.html#redis


Redis

If you’re using Redis as the broker, you can monitor the Celery cluster using the redis-cli command to list lengths of queues. Inspecting queues

Finding the number of tasks in a queue:

$ redis-cli -h HOST -p PORT -n DATABASE_NUMBER llen QUEUE_NAME

The default queue is named celery. To get all available queues, invoke:

$ redis-cli -h HOST -p PORT -n DATABASE_NUMBER keys \*

Note

Queue keys only exists when there are tasks in them, so if a key doesn’t exist it simply means there are no messages in that queue. This is because in Redis a list with no elements in it is automatically removed, and hence it won’t show up in the keys command output, and llen for that list returns 0. Also, if you’re using Redis for other purposes, the output of the keys command will include unrelated values stored in the database. The recommended way around this is to use a dedicated DATABASE_NUMBER for Celery, you can also use database numbers to separate Celery applications from each other (virtual hosts), but this won’t affect the monitoring events used by for example Flower as Redis pub/sub commands are global rather than database based.


Now, my Celery configuration (in Django) has the following relevant part:

CELERY_QUEUES = (
    Queue('default', Exchange('default'), routing_key='default'),
    Queue('email', Exchange('email'), routing_key='email'),
    Queue('haystack', Exchange('haystack'), routing_key='haystack'),
    Queue('thumbnails', Exchange('thumbnails'), routing_key='thumbnails'),
)

So I tried this:

$ redis-cli -n 0 -h ${MY_REDIS_HOST} -p 6379 llen haystack

(yes, celery is configured to use redis database number 0)

I tried my 4 queues, and I always get 0, when this is simply not possible. Some of these queues are usually very active, or my website wouldn't be working properly.

One key part of the documentation is that I can list the available queues, so I tried it:

$ redis-cli -n 0 -h ${MY_REDIS_HOST} -p 6379 keys \*

And I get about 20,000 lines of something like this:

celery-task-meta-b30fb605-d7b6-48db-b8cd-493458566876
celery-task-meta-e10ec56c-6601-420b-9f87-de6455968e76
celery-task-meta-14558a3a-1153-4f02-91f8-614bc29f6775
celery-task-meta-4c266854-512b-48af-8356-c786c507eb9e
celery-task-meta-e4ad4298-3d74-4986-8831-4c4d3c3e79f2
celery-task-meta-dfab0202-3975-46ce-9670-0d4cf3e278db
celery-task-meta-494fcb21-5995-495d-8980-0d8aa7edf0b8
celery-task-meta-345c4857-87f9-4e3f-8028-a6ef8cf93f5d
celery-task-meta-a4a48d00-68dc-4d30-87dd-869d2a20c347
celery-task-meta-d14fc394-6415-442b-8a5d-c9a4f37a9509

If I exclude all the celery-task-meta lines:

$ redis-cli -n 0 -h ${MY_REDIS_HOST} -p 6379 keys \* | grep -v celery-task-meta

I get this:

_kombu.binding.celeryev
_kombu.binding.default
_kombu.binding.thumbnails
_kombu.binding.email
unacked
_kombu.binding.celery.pidbox
_kombu.binding.haystack
unacked_index
_kombu.binding.reply.celery.pidbox

I tried to use the celery CLI to get the information, and this is some relevant output:

$ celery --app my-app inspect active_queues

-> celery@683a8e8bc84f: OK
    * {'name': 'thumbnails', 'exchange': {'name': 'thumbnails', 'type': 'direct', 'arguments': None, 'durable': True, 'passive': False, 'auto_delete': False, 'delivery_mode': None, 'no_declare': False}, 'routing_key': 'thumbnails', 'queue_arguments': None, 'binding_arguments': None, 'consumer_arguments': None, 'durable': True, 'exclusive': False, 'auto_delete': False, 'no_ack': False, 'alias': None, 'bindings': [], 'no_declare': None, 'expires': None, 'message_ttl': None, 'max_length': None, 'max_length_bytes': None, 'max_priority': None}
-> celery@bf11d4c3bd6f: OK
    * {'name': 'email', 'exchange': {'name': 'email', 'type': 'direct', 'arguments': None, 'durable': True, 'passive': False, 'auto_delete': False, 'delivery_mode': None, 'no_declare': False}, 'routing_key': 'email', 'queue_arguments': None, 'binding_arguments': None, 'consumer_arguments': None, 'durable': True, 'exclusive': False, 'auto_delete': False, 'no_ack': False, 'alias': None, 'bindings': [], 'no_declare': None, 'expires': None, 'message_ttl': None, 'max_length': None, 'max_length_bytes': None, 'max_priority': None}
-> celery@86151417b361: OK
    * {'name': 'default', 'exchange': {'name': 'default', 'type': 'direct', 'arguments': None, 'durable': True, 'passive': False, 'auto_delete': False, 'delivery_mode': None, 'no_declare': False}, 'routing_key': 'default', 'queue_arguments': None, 'binding_arguments': None, 'consumer_arguments': None, 'durable': True, 'exclusive': False, 'auto_delete': False, 'no_ack': False, 'alias': None, 'bindings': [], 'no_declare': None, 'expires': None, 'message_ttl': None, 'max_length': None, 'max_length_bytes': None, 'max_priority': None}
-> celery@9a5360a82f14: OK
    * {'name': 'haystack', 'exchange': {'name': 'haystack', 'type': 'direct', 'arguments': None, 'durable': True, 'passive': False, 'auto_delete': False, 'delivery_mode': None, 'no_declare': False}, 'routing_key': 'haystack', 'queue_arguments': None, 'binding_arguments': None, 'consumer_arguments': None, 'durable': True, 'exclusive': False, 'auto_delete': False, 'no_ack': False, 'alias': None, 'bindings': [], 'no_declare': None, 'expires': None, 'message_ttl': None, 'max_length': None, 'max_length_bytes': None, 'max_priority': None}

and

$ celery --app my-app inspect scheduled

-> celery@683a8e8bc84f: OK
    - empty -
-> celery@86151417b361: OK
    - empty -
-> celery@bf11d4c3bd6f: OK
    - empty -
-> celery@9a5360a82f14: OK
    - empty -

The command above seems to work well: if there are active tasks, the task is shown there, even tho in my copy/paste it says empty.

So, does anybody know what I might be doing wrong and why I can't get the real size of my queues?

Thanks!

1
  • Celery processes through tasks extremely fast (at least in our case), so it could be that you just got unlucky and queried Redis when there were no tasks in the queue. I would try making a task that sleeps for a long time in your test environment and call it in a loop so Celery can't process through them all before you query Redis.
    – Almenon
    Sep 8, 2022 at 20:03

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.