3

I have 13 node Cassandra cluster, now try to add 14 th node. each of 13 nodes holds about 1.5 TB data, now when I start new node, bootstrap process start without any problem and new node starts to get about 1Gbit data. But after some time it just stuck without any error in log files, nodetool netstats shows

/data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701   bytes(27%)  received  from idx:0/192.168.220.16
/data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671   bytes(38%)  received  from idx:0/192.168.220.22
/data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398   bytes(48%)  received  from idx:0/192.168.220.12
/data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610  bytes(85%)  received  from idx:0/192.168.220.18
/data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085   bytes(48%)  received  from idx:0/192.168.220.20

Seems that it looks as it should look, but /data/XXX/XXXX/tmp-la-XX-big-Data.db files does not exists on destination system, there is no network activity. Boostrap is paused and node is not joining to cluster, it status stales on UJ .

From time to time I see this in logfiles:

WARN  [STREAM-IN-/192.168.220.15] 2015-09-04 04:34:20,922 StreamSession.java:638 - [Stream #2bc66b80-52cf-11e5-bec6-75928743ca32] Retrying for following error
org.apache.cassandra.serializers.MarshalException: String didn't validate.
    at org.apache.cassandra.serializers.UTF8Serializer.validate(UTF8Serializer.java:35) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:91) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.cql3.ColumnIdentifier.<init>(ColumnIdentifier.java:58) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.db.composites.SimpleSparseCellNameType.fromByteBuffer(SimpleSparseCellNameType.java:83) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:162) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) [apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56) [apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-2.2.0.jar:2.2.0]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
ERROR [STREAM-IN-/192.168.220.15] 2015-09-04 04:34:20,927 StreamSession.java:518 - [Stream #2bc66b80-52cf-11e5-bec6-75928743ca32] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
    at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:90) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
INFO  [STREAM-IN-/192.168.220.15] 2015-09-04 04:34:22,048 StreamResultFuture.java:182 - [Stream #2bc66b80-52cf-11e5-bec6-75928743ca32] Session with /192.168.220.15 is complete
ERROR [Thread-12048] 2015-09-04 04:34:22,063 CassandraDaemon.java:182 - Exception in thread Thread[Thread-12048,5,main]
java.lang.RuntimeException: java.nio.channels.AsynchronousCloseException
    at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_79]
Caused by: java.nio.channels.AsynchronousCloseException: null
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205) ~[na:1.7.0_79]
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412) ~[na:1.7.0_79]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.7.0_79]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.7.0_79]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.7.0_79]
    at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:161) ~[apache-cassandra-2.2.0.jar:2.2.0]
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.0.jar:2.2.0]
    ... 1 common frames omitted

But server on which I can see this exception is not in list of staled stream servers.

I have tried to restart node, it starts bootstrapping, but stuck again.
We have : Cassandra 2.2.0 Datastx Community, Debian 7 64 bit, HotSpot JDK 1.7.0_79.

Have tried to disable autocompation during bootstrap on both source and destination nodes, repaired tables, but end up with same problem.

1
  • Were you able to scale cassandra?
    – sanjeev
    May 8, 2017 at 5:58

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.