4

Kafka v0.11.0.2

Kafka broker was running in PST timezone for about a month and then due to OS patch, Admin changed the timezone to EST and after restart, we saw replay of consumer message (consumer offset got reset to 0). We asked Admins to change the timezone back to PST, they took 3 days to change it. After changing the timezone back to PST, during broker restart we saw the bellow log messages (even for Topics and __consumer_offsets -> WARN Found a corrupted index file due to requirement failed:)

Since __consumer_offsets got corrupted when Kafka broker was restarted due to timezone changes, does it really reset the consumer offset to 0? We again got replay of messages after this message.

WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (/app/kafka_2.12-0.11.0.2/data/__consumer_offsets-21/00000000000000002076.index) has non-zero size but the last offset is 2076 which is no larger than the base offset 2076.}. deleting /app/kafka_2.12-0.11.0.2/data/__consumer_offsets-21/00000000000000002076.timeindex, /app/kafka_2.12-0.11.0.2/data/__consumer_offsets-21/00000000000000002076.index, and /app/kafka_2.12-0.11.0.2/data/__consumer_offsets-21/00000000000000002076.txnindex and rebuilding index... (kafka.log.Log)

How do we solve consumer offset not going back to zero (0)?

3 Answers 3

6

We had a similar issue when we were restarting our brokers. Turns out, for us, it was an issue with the systemd configuration that doesn't wait long enough to shutdown the Kafka server properly.

We have 350 GB of data on each broker with 150 topics and shutting down the Kafka server needs about 8 minutes. However, systemd was configured to wait only 90 seconds for the server to shutdown and then its force kills the server. When the server is restarted, it will end up having corrupted index file because its didn't shutdown properly. The fix was to set TimeoutStopSec=600 in kafka.service systemd configuration.

We summarised the issue and the fix in a blog post: https://blog.experteer.engineering/kafka-corrupted-index-file-warnings-after-broker-restart.html

1

Delete the index file.

This problem is based on the index cannot match the log file. This problem exists for long, the recovery way is to delete the index file, it will recreate.

1
  • 3
    While this suppresses the warning, its worth to understand what caused the corruption of the index file in the first place. Dec 18, 2019 at 9:57
0

I encount this problem,just delete the index file and start kafka:

cd /tmp/kafka-logs/dolphin-spider-google-book-bookinfo-0
# delete index file
rm -rf *

problem solved.Pay attention:this method would lost your current message.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.