1

I have installed Datastax AMI which is using EC2Snitch. The configurations are:

listen_address: private ip
broadcast_address: same as listen address
rpc_address: 0.0.0.0
broadcast_rpc: private ip
seeds: private ip

I have 2 such instances but I can't achieve consistency two. Gives alive:1 though both instances are running. I want to achieve consistency two from any client.

I tried this:

broadcast_rpc : public_ip  // same error
rpc_addess : public ip //cassandra wouldnt start

Would complain:

127.0.0.1:9042 is not running

What is the correct configuration? Nodes are in the same region, same rack. Nodetool gives both as up and running.

system.peers gives the private-ip in columns peer and roc_address and null in prefered-ip.

Cassandra.yaml:

cluster_name: 'logcluster'
num_tokens: 256
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
permissions_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
    - /mnt/cassandra/data
commitlog_directory: /mnt/cassandra/commitlog
disk_failure_policy: stop
commit_failure_policy: stop
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /mnt/cassandra/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "10.xxx.xx.xx7"
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 10.xxx.xx.xx5
start_native_transport: true
native_transport_port: 9042
start_rpc: true
rpc_address : 0.0.0.0
rpc_port: 9160
broadcast_rpc_address : 10.xxx.xx.xx5 
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
column_index_size_in_kb: 64
batch_size_warn_threshold_in_kb: 5
compaction_throughput_mb_per_sec: 16
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
phi_convict_threshold: 12
endpoint_snitch: Ec2Snitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
    internode_encryption: none
    keystore: conf/.keystore
    keystore_password: cassandra
    truststore: conf/.truststore
    truststore_password: cassandra
client_encryption_options:
    enabled: false
    keystore: conf/.keystore
    keystore_password: cassandra
internode_compression: all
inter_dc_tcp_nodelay: false
auto_bootstrap: false

Datacenter: us-east

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns (effective)  Host ID                       Rack
UN  10.xxx.xx.xx7   98.21 MB   256     53.9%             d5xxxx-0a59-xxxx-xxx-ab59xxxxx  1d
UN  10.xxx.xx.xx5  50.26 MB   256     46.1%             1xxxxff-xxx-xxx-xxx-13edxxxxxcf  1d
7
  • 1
    what are you using for your listen_address? rpc_address (thrift, 9160) and native (9042) are for you to connect to cluster with. The listen/broadcast addresses are whats used to communicate between nodes depending on snitch. Can you post your cassandra.yaml (removing comments so smaller be nice)? Jan 2, 2015 at 17:01
  • 1
    To help with troubleshooting, could you also include the output of nodetool status and the schema for the keyspace you're testing?
    – BrianC
    Jan 3, 2015 at 18:26
  • updated it in the question
    – mehnaazm
    Jan 4, 2015 at 14:51
  • iirc the Amazon nodes will show the private 10.x.y.z ip in ifconfig, the public IPs are not known by the node itself, instead these addresses are tagged on by NAT (network address translation). So the nodes should gossip on the 10.x.y.z network, as you can see from the nodetool output. Can you provide the error you see in cqlsh when you try to read / write at the desired consistency level. Also what replication factor is your keyspace, perhaps you could include the cql for your keyspace and table too? Thanks
    – markc
    Jan 13, 2015 at 18:57
  • Thanks @markc replication factor it is
    – mehnaazm
    Jan 14, 2015 at 9:54

1 Answer 1

1

The Amazon nodes will show the private 10.x.y.z ip in ifconfig, the public IPs are not known by the node itself, instead these addresses are tagged on by NAT (network address translation).

So the nodes should gossip on the 10.x.y.z network, as you can see from the nodetool output.

The best approach is to check the error you see in cqlsh when you try to read / write at the desired consistency level, this should give you a better handle on the issue. Also check the replication factor on your keyspace, for example:

CREATE KEYSPACE spam WITH replication = {
  'class': 'SimpleStrategy',
  'replication_factor': '2'
};

In the above example the replication factor is 2. If you have two nodes and you are reading with a consistency level of 1 there's a change you might not get the correct results. As a general rule of thumb it is recommended to always write at a higher consistency level than you read at. To set consistency in cqlsh use this syntax:

cqlsh> CONSISTENCY ONE;
Consistency level set to ONE.

Hope this helps

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.