13

If I have a RabbitMQ consumer that retrieves 100 messages in bulk, but then it crashes before it can mark those messages as processed, are those messages lost? I want every message in the queue to be processed at least once. What's the recommended approach to deal with consumers that crash before they've acknowledged messages?

Does RabbitMQ put them back on the queue somehow, or what do I need to do to make it happen?

2
  • if the consumer crashes, the messages will be put back to queue..
    – Jos
    Commented Feb 21, 2017 at 18:28
  • @redflar3 how will it know the consumer has crashed?
    – user779159
    Commented Feb 21, 2017 at 20:08

2 Answers 2

12

What's the recommended approach to deal with consumers that crash before they've acknowledged messages?

Let the rabbitmq do everything for you - messages that are not acknowledged are re-queued and will be delivered again to another (or even same) consumer.

Does RabbitMQ put them back on the queue somehow, or what do I need to do to make it happen?

See answer to first question. Simply don't acknowledge the messages before they're processed. This means also making sure that auto_ack flag is not set!

If I have a RabbitMQ consumer that retrieves 100 messages in bulk, but then it crashes before it can mark those messages as processed, are those messages lost?

See answer above - they are lost if they're automatically acknowledged.
Just a bit of reference, quoting from second tutorial:

If a consumer dies (its channel is closed, connection is closed, or TCP connection is lost) without sending an ack, RabbitMQ will understand that a message wasn't processed fully and will re-queue it. If there are other consumers online at the same time, it will then quickly redeliver it to another consumer. That way you can be sure that no message is lost, even if the workers occasionally die .

5
  • "messages that are not acknowledged are re-queued and will be delivered again" - how does it do this, does it re-queue the message if an acknowledgement hasn't arrived in some set amount of time?
    – user779159
    Commented Feb 21, 2017 at 20:11
  • basically if rmq detects that connection dropped it requeues all msgs for that consumer. Commented Feb 21, 2017 at 20:15
  • Can you please point me to the part of the documentation or wherever that this behavior is described?
    – user779159
    Commented Feb 21, 2017 at 21:17
  • 1
    I have edited my answer, there is a link to the tutorial Commented Feb 21, 2017 at 21:55
  • 2
    In this case do you know if the messages will get requeued to the beginning or end of the queue, meaning will they be picked up by the next consumer before or after everything else in the queue? Is it the same behavior as when you reject or nack a message with requeue set to true?
    – user779159
    Commented Feb 21, 2017 at 22:44
6

If the consumer has to acknowledge the message received, and it dies before it does that, the following things will happen:

  1. RabbitMQ broker will know that the consumer is not going to acknowledge, because the channel is closed.
  2. It will mark the message as undelivered.
  3. The order of message in the queue is preserved.
  4. The messages in the queue will be sent to the consumer, when it connects next time.

Point 3 will be void if you either have multiple consumers for the same queue or your RabbitMQ broker version is less than 2.7.0.

Hope it helps.

1
  • 3
    Is it possible to limit the number of times it will be returned to the queue? In case there is a recurring bug in the consumer that make him crashes repeatedly. @tbking
    – Avihay
    Commented May 9, 2022 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.