0

Elasticsearch 1.7.2 on CentOS

The question: When my nodes B and C went down, did I lose data?

3 node cluster: Nodes: A, B, C

A is master (was set up first, worked out that way). Relevant config (on all nodes, however what happened was the B lost network access and went down, and it turned out that C incorrectly was set to number_of_replicas: 1)

node.master: true
node.data: true
index.number_of_shards: 5
index.number_of_replicas: 2

On A, while those other two nodes were down, I notice that the "unassigned_shards" is 6. Since my shard count is 5, that implies to me that I have a problem:

# curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "elasticsearch-PROD-prod",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 6,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

Sure enough, on the shard list below, there is a primary shard (#1) that is UNASSIGNED

# curl -XGET http://localhost:9200/_cat/shards
index_v3_PROD 4 p STARTED    22578283 12.7gb 10.208.131.56 PROD-node-3a
index_v3_PROD 4 r UNASSIGNED                                   
index_v3_PROD 0 p STARTED    22572884 12.7gb 10.208.131.56 PROD-node-3a
index_v3_PROD 0 r UNASSIGNED                                   
index_v3_PROD 3 p STARTED    22579159 12.8gb 10.208.131.56 PROD-node-3a
index_v3_PROD 3 r UNASSIGNED                                   
index_v3_PROD 1 p UNASSIGNED                                   
index_v3_PROD 1 r UNASSIGNED                                   
index_v3_PROD 2 p STARTED    22580877 12.7gb 10.208.131.56 PROD-node-3a
index_v3_PROD 2 r UNASSIGNED                                   

Notice above that shard 1 is "p" and is UNASSIGNED. This looks scary to me!

I then used a reroute command to assign it over to A, which it did.

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{"commands" : [ {
              "allocate" : {
                  "index" : "index_v3_PROD", 
                  "shard" : 1, 
                  "node" : "PROD-node-3a", 
                  "allow_primary" : true
              }
            }
        ]
    }'

But shard 1 started at a very small size and then kind of grew (I think from new data being sent to ES). I have a strong feeling that shard 1 data was lost.

Can someone confirm whether shard 1 data looks suspect/lost (or not)?

5
  • I only see this "number_of_nodes" : 1 so only one node was available in the cluster. You said that only B went down. How come "number_of_nodes" is 1 then? Oct 28, 2015 at 0:42
  • @AndreiStefan Please re-read line 2 and 3 of OP Oct 28, 2015 at 23:24
  • What reroute command did you use exactly? Oct 28, 2015 at 23:48
  • @AndreiStefan OP enhanced w reroute command. What is the answer to the main question? (was data lost?) Oct 29, 2015 at 4:16
  • 1
    Yes, it was. A reroute command for a primary shard that has "allow_primary": true and that shard is not available will start the shard from scratch, empty. You should have made everything possible to bring that node back in the cluster. In your other post I asked your for the log files hoping to find out why the nodes are not able to join the cluster. Oct 29, 2015 at 6:41

1 Answer 1

0

Andrei posted this as a comment, not as an answer, so I will:

Yes, data was lost.

A reroute command for a primary shard that has "allow_primary": true and that shard is not available will start the shard from scratch, empty. You should have made everything possible to bring that node back in the cluster.


The footnote here is: We did not know what to do to bring back a cluster node. The other 2 nodes showed this status:

# curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "error" : "MasterNotDiscoveredException[waited for [30s]]",
  "status" : 503
}

We could not find any diag steps to use, and the logs showed us nothing useful. Two nodes both reported this issue (there are 3 total, so the only copy of the needed shard was on one of those two).

We verified all-way network communication, and rebooted the other nodes, but they did not attach to the cluster.

Ultimately, we set up a fresh 3 node cluster, and are being better about ensuring that every node has every shard present, so that the cluster could withstand losing 2 nodes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.