Hello all of you bright people,
We’re currently running a smallish 300 GB cluster in production on 5 nodes with around 30 mil docs. Everything works flawlessly except when a node really goes down (I mean like network or HW failure).
Generally when we lose a node the cluster becomes more or less completely unresponsive for a few minutes. Both regarding indexing and querying. This is of course, less than ideal as we have load 24/7.
I would really appreciate some help with understanding best practice settings to have robust cluster.
First goal for us is for the cluster to not become unresponsive in the event of a node crash. After reading everything I could find on the web I can't really understand if ES is designed to be unresponsive for ping_retries*ping_timeout seconds or if the cluster will continue to server query requests even during this time. Could anyone help me shed light on this?
Secondly in the event of a even worse failure where the cluster goes into red state, would it be possible to allow the cluster to still serve read/query requests?
I would be ever so grateful for anyone willing to help me understand how this works or what we would need to change to make our ES installation more robust.
I’ve included our config here:
cluster.name: clustername
node.name: nodename
path.data: /data
node.master: true
node.data: true
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.multicast.ping.enabled: false
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts: ["host1","host2","host3"]
bootstrap.mlockall: true
http.cors.enabled: true
index.number_of_shards: 10
action.disable_delete_all_indices: true
marvel.agent.exporter.es.hosts: ["marvel:9200"]
red
.yellow
from the start. Again, depending on the amount of data, shards started to be moved around, copies have been made from the replicas etc. This could take some time and corroborated with the operations you wanted to perform on the cluster these could have been put in the waiting queue, waiting for the administrative operations to finish. Now, another thing to look at: what did trigger the master to be unresponsive, was it long GC? If it was a long GC, did other nodes experience long old GCs? If so, it could be that you lost more than one nodes.