15

In my setup, I have a consumer group with three processes (3 instances of a service) that can consume from Kafka. What I've found to be happing is that the first node is receiving all of the traffic. If one node is manually killed, the next node picks up all Kafka traffic, but the last remaining node sits idle.

The behavior desired is that all messages get distributed evenly across all instances within the consumer group, which is what I thought should happen. As I understand, the way Kafka works is that it is supposed to distribute the messages evenly amongst all members of a consumer group. Is my understanding correct? I've been trying to determine why it may be that only one member of the consumer group is getting all traffic with no luck. Any thoughts/suggestions?

1 Answer 1

11

You need to make sure that the topic has more than one partition to be able to consume it in parallel. A consumer in a consumer group gets one or more allocated partitions from the broker but a single partition will never be shared across several consumers within the same group unless a consumer goes offline. The number of partitions a topic has equals the maximum number of consumers in a consumer group that can feed from a topic.

4
  • So each consumer has 4 threads and our topic has twelve partitions, so there is a thread per partition. Jun 25, 2014 at 20:43
  • Is it bad to set the partition 10000? If you don't know the amount of consumers?
    – Spidfire
    Oct 13, 2014 at 11:51
  • Partition coordination is currently done through Zookeeper and a higher number of partitions will put more pressure on your Zookeeper ensemble with slower re-balancing across consumers as a result. Another possible issue you might run into with too many partitions is the number of file handles on the brokers.
    – Lundahl
    Oct 13, 2014 at 18:21
  • 1
    @Lundahl, My topic has 3 partitions, However still just one consumer gets the message! where is the problem from? Dec 22, 2022 at 10:05

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.