I am trying to implement a way to randomly access messages from Kafka, by using KafkaConsumer.assign(partition), KafkaConsumer.seek(partition, offset). And then read poll for a single message.
Yet i can't get past 500 messages per second in this case. In comparison if i "subscribe" to the partition i am getting 100,000+ msg/sec. (@1000 bytes msg size)
I've tried:
- Broker, Zookeeper, Consumer on the same host and on different hosts. (no replication is used)
- 1 and 15 partitions
- default threads configuration in "server.properties" and increased to 20 (io and network)
- Single consumer assigned to a different partition each time and one consumer per partition
- Single thread to consume and multiple threads to consume (calling multiple different consumers)
- Adding two brokers and a new topic with it's partitions on both brokers
- Starting multiple Kafka Consumer Processes
- Changing message sizes 5k, 50k, 100k -
In all cases the minimum i get is ~200 msg/sec. And the maximum is 500 if i use 2-3 threads. But going above, makes the ".poll()", call take longer and longer (starting from 3-4 ms on a single thread to 40-50 ms with 10 threads).
My naive kafka understanding is that the consumer opens a connection to the broker and sends a request to retrieve a small portion of it's log. While all of this has some involved latency, and retrieving a batch of messages will be much better - i would imagine that it would scale with the number of receivers involved, with the expense of increased server usage on both the VM running the consumers and the VM running the broker. But both of them are idling.
So apparently there is some synchronization happening on broker side, but i can't figure out if it is due to my usage of Kafka or some inherent limitation of using .seek
I would appreaciate some hints of whether i should try something else, or this is all i can get.