What is the difference between LEO and HW
in Replica ( Leader Replica
)?
Will they contain the same number? I can understand HW is the last committed message offset
.
When LEO will be updated and how?
What is the difference between LEO and HW
in Replica ( Leader Replica
)?
Will they contain the same number? I can understand HW is the last committed message offset
.
When LEO will be updated and how?
The high watermark indicated the offset of messages that are fully replicated, while the end-of-log offset might be larger if there are newly appended records to the leader partition which are not replicated yet.
Consumers can only consume messages up to the high watermark.
See this blog post for more details: http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
Let's start with one of the most popular watermark definition that could be find on Google
The high watermark is the offset of the last message that was successfully copied to all of the log’s replicas
I was not very well convinced of the above definition and going deeper in my research I found this nice picture:
What was wrong then? The stuck follower at the extreme right in the pic didn't have the fourth message logged. Maybe the first definition Google found was not complete and what the author did actually intend was: "The high watermark is the offset of the last message that was successfully copied to all of the log’s in-sync replicas"
Guided by this intuition I found this article providing details about how the WM is computed together with code.
I found the WM definition reported much more precise:
High watermark is calculated as the minimum LEO across all the ISR of this partition, and it grows monotonically.
This answer together with the code provided confirmed the intuition I had.
Summing up I think the detailed definition of the watermark shows what is the difference between LEO and WM. The latest committed offset and LEO might coincide with high water mark for in sync follower but may very well not for Leader as shown by the example in the first linked image.