1

We are about to pick Google Cloud Pub/Sub as our MQ. Some of the messages need to be dispatched in low latency durations like <100-200ms.

I read somewhere that in high volumes of messages, it performs well, but for lesser amounts like hundreds of messages per second, Pub/Sub may yield high latencies.

And in that previous question, the topic is not clear as well. what's google cloud pub/sub latency (year 2016)

Are there any defined metrics about Pub/Sub latencies and relationship with the volume of messages?

2

Cloud Pub/Sub was designed with high volumes of messages in mind, so it's expected for it to struggle sometimes if the message throughput is too low. This has to do with how Pub/Sub balances the pull requests between listeners.

Back to your question, there are no defined metrics regarding this behaviour, also not minimal latency or a magic number of messages that you can say it's considered "high load". In reality it depends on factors like the number of workers, batch size, processing time of each message, etc.

I've found this other StackOverflow answer that talks a bit more about a valid use case for Pub/Sub (messages in the order of thousands per second) and why high loads are recommended for Cloud Pub/Sub.

Edit:

Expanding on this topic, adding an asynchronous layer to the Pub/Sub implementation can help improve its performance, including latency in low message volume scenarios. Spotify, when working with Pub/Sub, realized the need for this and created their own Async Cloud Pub/Sub Client. They now recommend using Google's java client, but is worth taking this information into account

  • Thank you @guillermo-cacheda, your answer matches that I found on the Internet. A small question regarding the high volume: When we say 1000 messages/second, do we mean all 1000 on a single topic or this is the total amount of messages published to Pub/Sub? – Erdem A Nov 8 '18 at 17:20
  • @ErdemA given that the issues come from balancing between listeners, I would say the hypothetical 1000 messages/second refer to a single topic. The example I linked refers to a single topic, as far as I know. Remember that 1000 messages are not necessarily considered "high volume". Also, I edited more info on my answer. I believe it's useful and expands on the information I provided the first time. – Guillermo Cacheda Nov 8 '18 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.