6

I have a 5 node Cassandra 2.0.7 cluster, each node has 4 HDDs. Recently one of these HDDs on node3 had failed and was replaced by a new shiny empty drive. After the replacement cassandra on this node was unable to start with this exception:

 INFO [main] 2014-06-02 12:45:17,232 ColumnFamilyStore.java (line 254) Initializing system.paxos
 INFO [main] 2014-06-02 12:45:17,236 ColumnFamilyStore.java (line 254) Initializing system.schema_columns
 INFO [SSTableBatchOpen:1] 2014-06-02 12:45:17,237 SSTableReader.java (line 223) Opening /mnt/disk2/cassandra/system/schema_columns/system-schema_columns-jb-310 (25418 bytes)
 INFO [main] 2014-06-02 12:45:17,241 ColumnFamilyStore.java (line 254) Initializing system.IndexInfo
 INFO [main] 2014-06-02 12:45:17,245 ColumnFamilyStore.java (line 254) Initializing system.peers
 INFO [SSTableBatchOpen:1] 2014-06-02 12:45:17,246 SSTableReader.java (line 223) Opening /mnt/disk3/cassandra/system/peers/system-peers-jb-25 (20411 bytes)
 INFO [main] 2014-06-02 12:45:17,253 ColumnFamilyStore.java (line 254) Initializing system.local
 INFO [SSTableBatchOpen:1] 2014-06-02 12:45:17,254 SSTableReader.java (line 223) Opening /mnt/disk3/cassandra/system/local/system-local-jb-35 (80 bytes)
 INFO [SSTableBatchOpen:2] 2014-06-02 12:45:17,254 SSTableReader.java (line 223) Opening /mnt/disk3/cassandra/system/local/system-local-jb-34 (80 bytes)
 ERROR [main] 2014-06-02 12:45:17,361 CassandraDaemon.java (line 237) Fatal exception during initialization
  org.apache.cassandra.exceptions.ConfigurationException: Found system keyspace files, but they couldn't be loaded!
    at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:532)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:233)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:552)

Because of cassandra node being unable to start, I cannot use nodetool repair.

The only way I see to recover the node is to remove all data and bootstrap it from nearly bare metal. Is there a shorter way to recover in a typical HDD failure scenario?

2 Answers 2

20

Fixed the issue by these steps:

  • physically removed files related to system keyspace: cassandra was able to start and recreated it, but without any metadata about other keyspaces.

  • ran nodetool resetlocalschema, which synchronized keyspace schema from other nodes.

1
  • 4
    This worked for me as well. For the removal I used: rm -rf /var/lib/cassandra/data/system/ (CentOS/RHEL)
    – TomS
    Aug 7, 2015 at 1:51
0

Another solution I found was to simply delete your data directory within your parent Cassandra directory. Upon starting Cassandra again it should recreate your namespace.

1
  • But in case if you have multi-HDD setup, single HDD failure can render your node unusable. The question was how to recover node without dropping all the existing data on non-failed HDDs.
    – shutty
    Apr 6, 2016 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.