4

I had an unexpected shutdown of Mac OS X while running apache-cassandra 3.3. When restarting cassandra in a new bootload, I got the following error:

INFO  15:25:15 Replaying /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351786.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351787.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461942807785.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461943004975.log
ERROR 15:25:15 Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 542295 of commit log /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351786.log, with bad position but valid CRC
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:611) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.SegmentReader$SegmentIterator.computeNext(SegmentReader.java:105) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.SegmentReader$SegmentIterator.computeNext(SegmentReader.java:78) [apache-cassandra-3.5.jar:3.5]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) [guava-18.0.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) [guava-18.0.jar:na]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:399) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.jar:3.5]

I searched on stack overflow and google and did not find posts of this same problem. I found issue https://issues.apache.org/jira/browse/CASSANDRA-9749, which is related, but this one is marked resolved and is related to continuing execution after an problem is detected rather than failing with an exception.

What do I do in this situation? Is there any recovery path (assuming no snapshots avaialble)? Any way to truncate the commit log at the offending record? Any tool available to fix the bad header (e.g. delete the record)?

Thanks.

3 Answers 3

9

I also faced this exception sometimes when playing with my local Cassandra on Mac OS.

If you don't care about loosing the data in the commit logs or if you have a backup somewhere, just stop Cassandra, clean the commitlog folder and restart.

I'm afraid that there is no other alternative. By the way, are you deploying Cassandra on Mac OS for production ?

  • If yes, you should know that Mac OS is not officially supported
  • If it's not for production then who care, just clean the commitlog and restart
1
  • i am new to cassandra...please tell me how can i start cassandra
    – H Raval
    Apr 7, 2017 at 12:11
2

I faced similar kind of problem while running Cassandra on Windows 7 (local machine) environment. The exception I found from log is

ERROR 07:14:35 Exiting due to error while processing commit log during initialization. org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not read commit log descriptor in file C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1470969948498.log at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:616) [apache-cassandra-3.7.0.jar:3.7.0] at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:378) [apache-cassandra-3.7.0.jar:3.7.0]

As it was just a test setup I deleted that particular commit log file and restarted it from windows service manager and it started perfectly. I'm not sure what to do if it happens on production environment.

0

The second time I get this on my single node developer system. Seems to be able to happen if jvm is killed unexpectedly. Clearing the commitlog catalog solved it. Using Windows 10.

What worries me is that I get these error issues even if no update were made to the database. Doesn't cassandra flush on a regular basis. I do not want to have issues like this in production just because a server crash or similar, even if I have replica nodes.

In the stdout log, the last part is

INFO  06:17:39 Replaying C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812251.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471411951134.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471454506802.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471532812678.log
ERROR 06:17:39 Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not read commit log descriptor in file C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:611) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:373) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.4.0.jar:3.4.0]
1
  • My config says that commitlog are synced every 10s seconds, so how can a file be corrupt unless a crash occurs within these 10 seconds?
    – mattias
    Aug 19, 2016 at 6:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.