1

I am trying to implement a way to randomly access messages from Kafka, by using KafkaConsumer.assign(partition), KafkaConsumer.seek(partition, offset). And then read poll for a single message.

Yet i can't get past 500 messages per second in this case. In comparison if i "subscribe" to the partition i am getting 100,000+ msg/sec. (@1000 bytes msg size)

I've tried:

  1. Broker, Zookeeper, Consumer on the same host and on different hosts. (no replication is used)
  2. 1 and 15 partitions
  3. default threads configuration in "server.properties" and increased to 20 (io and network)
  4. Single consumer assigned to a different partition each time and one consumer per partition
  5. Single thread to consume and multiple threads to consume (calling multiple different consumers)
  6. Adding two brokers and a new topic with it's partitions on both brokers
  7. Starting multiple Kafka Consumer Processes
  8. Changing message sizes 5k, 50k, 100k -

In all cases the minimum i get is ~200 msg/sec. And the maximum is 500 if i use 2-3 threads. But going above, makes the ".poll()", call take longer and longer (starting from 3-4 ms on a single thread to 40-50 ms with 10 threads).

My naive kafka understanding is that the consumer opens a connection to the broker and sends a request to retrieve a small portion of it's log. While all of this has some involved latency, and retrieving a batch of messages will be much better - i would imagine that it would scale with the number of receivers involved, with the expense of increased server usage on both the VM running the consumers and the VM running the broker. But both of them are idling.

So apparently there is some synchronization happening on broker side, but i can't figure out if it is due to my usage of Kafka or some inherent limitation of using .seek

I would appreaciate some hints of whether i should try something else, or this is all i can get.

1 Answer 1

6

Kafka is a streaming platform by design. It means there are many, many things has been developed for accelerating sequential access. Storing messages in batches is just one thing. When you use poll() you utilize Kafka in such way and Kafka do its best. Random access is something for what Kafka don't designed.

If you want fast random access to distributed big data you would want something else. For example, distributed DB like Cassandra or in-memory system like Hazelcast.
Also you could want to transform Kafka stream to another one which would allow you to use sequential way.

3
  • Thanks, i can understand that. And it was also what i expected that a single Consumer would be ~1000x slower consuming random messages, than when consuming batches (>100,000 msg/sec vs <200 msg/sec). But i would have expected that with more resources (CPU/Memory) this would also scale... That is more partitions/higher replication factor/something else (i haven't tried more replicas yet)... but it doesn't.
    – vlast3k
    Commented Apr 13, 2019 at 2:30
  • @vlast3k If you read design doc you could see that 1. sequential access benefits from disk and system cache and even HDD are fast enough in that case. But with random access it would fail. So the botleneck is in IO, not in RAM or CPU. 2. Increasing replication factor doesn't matter since only 1 replica send data to the consumer. 3. Increasing number of partitions would lead to bigger network traffic and stil IO is bottleneck. 4. Messages are stored in blocks so Kafka have to read many messages when send you only one.
    – ADS
    Commented Apr 13, 2019 at 13:39
  • thanks for the link. I didn't look into this part of the documentation in too much detail yet. Perhaps this is also the reason why i get the same performance with different sized messages. At the end if the effort Kafka does to retrieve the messages from the store is one and the same - then size does not matter. But yeah, i realize that the large amount of messages read sequentially might be the result of all the optimizations, while the random read is rather the norm
    – vlast3k
    Commented Apr 15, 2019 at 1:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.