40

I am using kafka 0.10.1.1 and confused with the following 3 properties.

heartbeat.interval.ms
session.timeout.ms
max.poll.interval.ms

heartbeat.interval.ms - This was added in 0.10.1 and it will send heartbeat between polls. session.timeout.ms - This is to start rebalancing if no request to kafka and it gets reset on every poll. max.poll.interval.ms - This is across the poll.

But, when does kafka starts rebalancing? Why do we need these 3? What are the default values for all of them?

Thanks

2
  • We solved this problem by increasing session.timeout.ms and decreasing max.poll.records . It works but we're still looking for better solutions..
    – Tony Tang
    Commented Jul 11, 2017 at 9:04
  • What is "this" problem? @TonyTang
    – kev
    Commented Oct 5, 2022 at 22:20

3 Answers 3

67

Assuming we are talking about Kafka 0.10.1.0 or upwards where each consumer instance employs two threads to function. One is user thread from which poll is called; the other is heartbeat thread that specially takes care of heartbeat things.

session.timeout.ms is for heartbeat thread. If coordinator fails to get any heartbeat from a consumer before this time interval elapsed, it marks consumer as failed and triggers a new round of rebalance.

max.poll.interval.ms is for user thread. If message processing logic is too heavy to cost larger than this time interval, coordinator explicitly have the consumer leave the group and also triggers a new round of rebalance.

heartbeat.interval.ms is used to have other healthy consumers aware of the rebalance much faster. If coordinator triggers a rebalance, other consumers will only know of this by receiving the heartbeat response with REBALANCE_IN_PROGRESS exception encapsulated. Quicker the heartbeat request is sent, faster the consumer knows it needs to rejoin the group.

Suggested values:
session.timeout.ms : a relatively low value, 10 seconds for instance.
max.poll.interval.ms: based on your processing requirements
heartbeat.interval.ms: a relatively low value, better 1/3 of the session.timeout.ms

5
  • 1
    So, rebalance will happen if session.timeout.ms is met or max.poll.interval.ms. Whichever is met first. But, session.timeout.ms will be updated only when the poll method is called. Say, session.timeout.ms is configured to 2 secs and max.poll.interval.ms is configured to 4 ms. But, say processing is taking 3 secs and hence the poll method will be invoked after 3 secs, but session.timeout.ms is already reached and hence the rebalance would have started. What is the use of max.poll.interval.ms here? And, heartbeat.interval.ms is only for letting other consumers know asap. Commented May 16, 2017 at 4:55
  • I did not see any valid scenarios where poll interval is set such low and much less than session timeout. In fact, introducing max.poll.interval.ms and setting it to a relatively large value is just a result of giving consumer more time to process messages but not sacrificing the failure detection time.
    – amethystic
    Commented May 16, 2017 at 12:41
  • 2
    Anyway, once the session.timeout.ms is reached, the rebalance will be triggered. Say, it takes 10 secs for processing and the session.timeout.ms is been configured to 8 secs. Since it takes 10 secs for processing, the next poll method will be invoked after 10 secs, and hence the rebalance will be started as the session.timeout.ms is 8 secs. Even we set the max.poll.interval.ms to 60 secs, what is the use of it as the rebalance has already started based on the session.timeout.ms which is lower than max.poll.interval.ms. Commented May 16, 2017 at 13:58
  • 1
    @user1578872 This is good description given : cwiki.apache.org/confluence/display/KAFKA/…
    – Vinit89
    Commented Sep 12, 2017 at 6:49
  • so re-balance will happen if any one these conditions are met - (1) no heart beat between session.timeout.ms (2) no poll between max.poll.interval.ms. Plz confirm. Or (though this should never be a case) when max.poll.interval.ms condition is met, session.timeout.ms is ignored as the former indicates that the consumer is processing and healthy - so re-balance is NOT triggered.
    – samshers
    Commented Jan 15, 2023 at 21:21
5

session.timeout.ms is closely related to heartbeat.interval.ms.

heartbeat.interval.ms controls how frequently the KafkaConsumer poll() method will send a heartbeat to the group coordinator, whereas session.timeout.ms controls how long a consumer can go without sending a heartbeat.

Therefore, those two properties are typically modified together. heatbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout value. So if session.timeout.ms is 3 seconds, heartbeat.interval.ms should be 1 second.

max.poll.interval.ms - The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member

1
  • so re-balance will happen if any one these conditions are met - (1) no heart beat between session.timeout.ms (2) no poll between max.poll.interval.ms. Plz confirm. Or (though this should never be a case) when max.poll.interval.ms condition is met, session.timeout.ms is ignored as the former indicates that the consumer is processing and healthy - so re-balance is NOT triggered.
    – samshers
    Commented Jan 15, 2023 at 21:19
4

just make them more clear, a heartbeat thread(along with user thread which invokes Poll function in the same process) will send heartbeat to coordinator every "heartbeat.interval.ms" time, and the coordinator will mark the consumer in the user thread as dead if it exceeds "session.timeout.ms" or "max.poll.interval.ms".

1
  • so re-balance will happen if any one these conditions are met - (1) no heart beat between session.timeout.ms (2) no poll between max.poll.interval.ms. Plz confirm. Or (though this should never be a case) when max.poll.interval.ms condition is met, session.timeout.ms is ignored as the former indicates that the consumer is processing and healthy - so re-balance is NOT triggered.
    – samshers
    Commented Jan 15, 2023 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.