79

Suppose I had five nodes in a cluster and I had to remove two nodes at run time. So how can it be done without affecting the indices?

I had a continuous stream of data coming at nearly 10 Gb/hour which is getting indexed continuously.

Would rebalancing be a help in this?

2
  • why is solr tagged in this question? I guess this is only elasticsearch related question. Please don't use wrong tags. This might mislead users to look into irrevlevant question. Jun 26, 2013 at 7:37
  • 10 gigabit/hour or 10 gigabyte/hour? Dec 29, 2022 at 1:07

2 Answers 2

149

You can decommission a node by telling the cluster to exclude it from allocation. (From the documentation here)

curl -XPUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
   }
}';echo

This will cause Elasticsearch to allocate the shards on that node to the remaining nodes, without the state of the cluster changing to yellow or red (even if you have replication 0).

Once all the shards have been reallocated you can shutdown the node and do whatever you need to do there. Once you're done, include the node for allocation and Elasticsearch will rebalance the shards again.

8
  • 37
    Also "cluster.routing.allocation.exclude._name" and "cluster.routing.allocation.exclude._id" can be used to decommission by node name and node id. Needed this in a situation when due to incorrect configuration (/etc/hosts & elasticsearch.yml) all nodes got the same (published) IP 127.0.1.1. Nov 19, 2014 at 16:28
  • 3
    How do I check if it is OK to shut down the decommissioned node? May 14, 2015 at 14:02
  • 17
    You could use curl -XGET 'http://ES_SERVER:9200/_cluster/health?pretty'; if there are no nodes relocating, then the excluded node must have been drained and safe to shut down. Another option is to check curl -XGET 'http://ES_SERVER:9200/_nodes/NODE_NAME/stats/indices?pretty' to see if there are no documents left on the node.
    – towr
    May 14, 2015 at 20:21
  • 14
    Multiple _ip records can be comma separated, ex: "cluster.routing.allocation.exclude._ip" : "10.0.0.1,10.0.0.2" Oct 30, 2015 at 20:54
  • 1
    @TomMiller - It doesn't matter. if you take it out of the routing allocation before removing it from the load balancer it'll still handle queries and return results.
    – Paul Lemke
    Dec 22, 2016 at 16:10
11

To remove an Elasticsearch node from the cluster. Just run the following command:

curl -XPUT P.P.P.P:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "X.X.X.X"
   }
}';echo

Here P.P.P.P is the private IP address of the master node. You may also use the localhost if Elasticsearch is running on localhost. X.X.X.X is the private IP address of the node to be removed from the cluster.

This command will give acknowledgement true if the node is accepted to be removed and the data relocation will start. Check if the data relocation is over and the node doesn't have any shards left on it. Then stop the elasticsearch process and stop/terminate the instance.

The commands to check data relocation and shards left can be found in this article.

2
  • I do not think this is accurate. At least on v1.7.5 if you exclude a node it will not get NEW shards allocated to it but existing shards are not effected. Nov 15, 2016 at 15:31
  • On v1.7.1 excluding a node will affect existing shards.
    – Yani
    Feb 12, 2021 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.