Elasticsearch 1.7.2 on CentOS
The question: When my nodes B and C went down, did I lose data?
3 node cluster: Nodes: A, B, C
A is master (was set up first, worked out that way). Relevant config (on all nodes, however what happened was the B lost network access and went down, and it turned out that C incorrectly was set to number_of_replicas: 1)
node.master: true
node.data: true
index.number_of_shards: 5
index.number_of_replicas: 2
On A, while those other two nodes were down, I notice that the "unassigned_shards" is 6. Since my shard count is 5, that implies to me that I have a problem:
# curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
"cluster_name" : "elasticsearch-PROD-prod",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 4,
"active_shards" : 4,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 6,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
Sure enough, on the shard list below, there is a primary shard (#1) that is UNASSIGNED
# curl -XGET http://localhost:9200/_cat/shards
index_v3_PROD 4 p STARTED 22578283 12.7gb 10.208.131.56 PROD-node-3a
index_v3_PROD 4 r UNASSIGNED
index_v3_PROD 0 p STARTED 22572884 12.7gb 10.208.131.56 PROD-node-3a
index_v3_PROD 0 r UNASSIGNED
index_v3_PROD 3 p STARTED 22579159 12.8gb 10.208.131.56 PROD-node-3a
index_v3_PROD 3 r UNASSIGNED
index_v3_PROD 1 p UNASSIGNED
index_v3_PROD 1 r UNASSIGNED
index_v3_PROD 2 p STARTED 22580877 12.7gb 10.208.131.56 PROD-node-3a
index_v3_PROD 2 r UNASSIGNED
Notice above that shard 1 is "p" and is UNASSIGNED. This looks scary to me!
I then used a reroute command to assign it over to A, which it did.
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{"commands" : [ {
"allocate" : {
"index" : "index_v3_PROD",
"shard" : 1,
"node" : "PROD-node-3a",
"allow_primary" : true
}
}
]
}'
But shard 1 started at a very small size and then kind of grew (I think from new data being sent to ES). I have a strong feeling that shard 1 data was lost.
Can someone confirm whether shard 1 data looks suspect/lost (or not)?
"number_of_nodes" : 1
so only one node was available in the cluster. You said that only B went down. How come"number_of_nodes"
is1
then?reroute
command did you use exactly?reroute
command for a primary shard that has"allow_primary": true
and that shard is not available will start the shard from scratch, empty. You should have made everything possible to bring that node back in the cluster. In your other post I asked your for the log files hoping to find out why the nodes are not able to join the cluster.