0

I have few questions on Apache Kafka.

  1. Can a single partition be assigned to more than one consumer from the same group?
  2. Where is the offset stored? Is it in the partition or at the consumer.
  3. Just like the producer always post the record to the lead partition and the records gets replicated to other partitions, Does Kafka consumer reads the data from the lead partition?
  4. Lets say, that a consumer is reading from a partition and the consumer is running a long process. In this case, the rate at which the producer is updating the partition will be faster than the rate at which the consumer is consuming from the same partition. Is there a way we can speed up the consumption from that partition?
  5. Can we create a checkpoint in the commit log on the partition so that the consumer can start processing from that specific checkpoint? This would be useful, if I want to perform the audit from a specific checkpoint onward?

1 Answer 1

1

Can a single partition be assigned to more than one consumer from the same group?

No, one partition can be consumed at most from one consumer within the same consumer group as described here: "This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group."

Where is the offset stored? Is it in the partition or at the consumer.

The offsets for each consumer group is stored in an internal kafka topic called __consumer_offsets as described here: "The coordinator of each group is chosen from the leaders of the internal offsets topic __consumer_offsets, which is used to store committed offsets."

Just like the producer always post the record to the lead partition and the records gets replicated to other partitions, Does Kafka consumer reads the data from the lead partition?

Yes it does. The leader partition is the only "client-facing" partition as described here: "'leader' is the node responsible for all reads and writes for the given partition.".

EDIT:

Is there a way we can speed up the consumption from that partition?

The measure to speed up consumption is to increase the partitions of the topic so you can have more consumer threads reading from that topic and process the data in parallel. At the same time you need to make sure that your data is evenly distributed accross partitions.

4
  • I have added a follow up question on the consumer. Can you please answer
    – zilcuanu
    Jul 28, 2020 at 16:57
  • Is there a slack channel where I can involve in Kafka discussions?
    – zilcuanu
    Jul 28, 2020 at 17:19
  • Not sure about this. At least I am not aware of. But there should be mailing list available. Regarding your 5th question. Yes, this is possible but highly depends on your client API. If using Java KafkaConsumer you can make use of seek method. More details are given in the JavaDocs on KafkaConsumer Jul 28, 2020 at 17:22
  • @zilcuanu Confluent has a public Slack channel for Kafka questions. You can also speed up consumption by playing with buffer and timeout settings, and reducing the number of records processed in the poll loop Jul 28, 2020 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.