1

I have ELK stack with Elasticsearch, Logstash and kibana installed on 3 different instances.

Now I want to make 3 node cluster of Elasticsearch.

I will make one node as master and 2 data nodes.

I want to know in logstash config

elasticsearch {
    hosts => "http://es01:9200"

Which address I need to enter there master node or data node. and also if I have 3 master nodes then which address I need to write there.

similarly in kibana , I use

elasticsearch.url: es01:9200

In cluster env which url I need to use?

1 Answer 1

3

In general, the answer depends on your cluster data size and load.
Nevertheless, I'll try to answer your questions assuming the master node is not a data eligible node as well. This means it only takes care for cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes. For this purposes, it is very recommended to have your master node as stable and less loaded as possible. So, in your logstash config I would put the addresses of your two data nodes as follows:

elasticsearch{
    hosts => ["http://es01:9200", "http://es02:9200"]
}

This confirmation maximize performance and fault tolerance as your master do not contain data and if one node failes it will continue to work with the other.

Please note that it is very recommended to have at least 3 master eligible nodes configured in Elasticsearch clusters since if you are loosing the (only) master node you loose data. 3 is to avoid split brain

Regarding kibana, since all nodes in the cluster "knows" each other. You basically can put any address in the cluster. But, for the same reasons as above it is recommended to fill one of your data nodes addresses.

For further reading, please refer to this documentation.

Hope I have managed to help!

5
  • Thanks for clearing doubt. How does each node know about each other. I mean in each data node what config i need to write so that they know what are master nodes and data nodes available, provided i don't have autodiscovery because of firewall thing
    – Karl
    Jul 8, 2016 at 2:51
  • There are many ways to connect nodes in the cluster depending on your language of choice, desired method and environment. Assume for simplicity that all configuration done through elasticsearch.yaml and you are running on non cloud environment. On each node you'll need to write the following in elasticsearch.yaml: hosts: ["node1:9300","node2:9300"]. You also need to have Elasticsearch transport port open in your firewall. Restart Elasticsearch on each node after setting this. It is enough to put two nodes which are located in the cluster, all other nodes will be discovered automatically. Jul 8, 2016 at 4:44
  • Please also note that if your firewall do not permit 9300 you can use an other transport port and then configure field transport.tcp.port in elasticsearch.yml to an other port which your firewall allows. ex. 8080: transport.tcp.port: 8080 Jul 8, 2016 at 5:42
  • Thanks for your help. I have made 3 master nodes , 4 data nodes and 2 client nodes. So in the logstash do i need to enter all client nodes and they will know where to load balance?
    – Karl
    Jul 10, 2016 at 0:01
  • Yes, the clients may serve as coordinators for your cluster and load balance request from Logstash. Jul 11, 2016 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.