1

I've a relatively standard Publish/Subscribe system with the following requirements:

  • Each message contains number (1 to few tens, but typically no more than 5) of topics.
  • Each consumer subscribe to number of topics (typically: 3-10).
  • There are thousands of topics, hundreds of consumers and couple of producers.
  • The system has to scale 2000 messages per second.
  • On occasion, consumer can change the topics the are interested in.
  • The requirement is of course that every message will reach its subscriber exactly once (even if it contain several keywords subscribed by the consumer).

I'm using RabbitMQ, but I think the question is general enough.

None of the exchange types that Rabbit has directly support this model. It could be really cool if RabbitMQ supported Regular Expression binding to queue, but I don't think this exists.

I was thinking about a scheme where if client is interested in key1, key2, and key3, it will create a binding to #.key1.#, #.key2.#, and #.key3.#, and a message with key1 and key4 has a routing key of: .key1.key4. I wonder if this is the correct paradigm or whether a more canonical approach exists. How expensive is the routing? How expensive is changing the binding when consumer change their subscription? What happen to messages already in the queue after the binding has change.

1 Answer 1

1

Short answer:

Routing model is good enough. Don't worry about rabbitmq performance (especially with 2k/s message flow). Once message get into queue it will still there unless it get consumed or died (due to ttl or length limit).

Long answer:

I was thinking about a scheme where if client is interested in key1, key2, and key3, it will create a binding to #.key1.#, #.key2.#, and #.key3.#, and a message with key1 and key4 has a routing key of: .key1.key4. I wonder if this is the correct paradigm or whether a more canonical approach exists.

The idea is good enough. If you want, you may also experiment with headers exchanges if you are sure you need their power (if you don't - stick with topic).

How expensive is the routing?

Don't worry about topic exchanges routing performance, it is fast enough and well-optimized in RabbitMQ broker internals.

For details read RabbitMQ blog posts with 'performance' tag, especially Very fast and scalable topic routing – part 1 and Very fast and scalable topic routing – part 2.

How expensive is changing the binding when consumer change their subscription?

It's a trivial operation which doesn't make any performance impact at all.

What happen to messages already in the queue after the binding has change.

Once message get into queue it will still there unless you get it from there with basic.get or basic.consume methods or message get dead lettered:

  • The message is rejected (basic.reject or basic.nack) with requeue=false,
  • The TTL for the message expires; or
  • The queue [length limit is exceeded.

P.S.:

There might be some caveats like synchronization delay (when you get into HA or clustering), network failure, etc. that introduce additional conditions under which message might be lose (when it consumed without confirmation and was removed from queue but not delivered to consumer due to, for example, consumer application failure) or doubled to another (during dead lettering), but these are very special cases that one will know when get into it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.