I need to read the latest messages from kafka topic that contains 3 partitions. I need to read the same data from 5 different clients, first I tried to work with the Simple Consumer but it is not working with multiple partitions. The aim is to have the partitions handling of the Group consumer and also get the same msgs to all clients.
You can use different consumer group names for each client.
Every Kafka consumer is part of consumer group. The way it works is that all messages are consumed by every consumer group. But it will be consumed by only one consumer in that group. You can read this for more details
-
Yep, that's exactly what I did, this is how the command line consumer works. My only concern is what about the "garbage" that Kafka saves on Zookeeper. Every time a consumer group is being created Kafka also creates a znode with the consumer group details- like what is the last offset the group fetched message from. – 15412s Jan 16 '15 at 11:31
-
You are correct. For every new consumer, data is stored in zookeeper. I think you shouldn't be worried much about these zNodes as these are internal to Kafka and also they will have very less data. – puneet Jan 17 '15 at 13:08
-
What is the use to have a multiple consumer in same consumer group, if only one consumer is going to consume message in that group – Gnana Jan 4 '16 at 22:38
-
One partitions data is read by one consumer. Other consumers can read from other partitions. Also, one consumer can read from multiple partitions. This is useful in following ways 1. Distributing load: Lets say you have 1000 partitions. If you run have only consumer, all processing is done by it. Having multiple consumer distributes load 2. High availability: If one consumer goes down, other consumers start reading its partition data. – puneet Jan 6 '16 at 4:36
You should just use different consumer group names for different clients.
Consider this: you have a topic with 3 partitions. Client 1 has a running consumer (or multiple consumers whatever) in group 1 and gets messages from all partitions. Client 2 has a consumer in group 2 and so on..
This way you may have 2 consumers in group 1 that consume messages for client 1, and 3 consumers in group 2 that consume messages for client 2 or any other configurations you may imagine.