0

I have two docker nodes running in swarm like below. The second node i promoted to work as manager.

imb9cmobjk0fp7s6h5zoivfmo * Node1 Ready Active Leader 19.03.11-ol a9gsb12wqw436zujakdpbqu5p Node2 Ready Active Reachable 19.03.11-ol

This works fine when leader node goes to drain/pause. But as part of my test i have stopped the Node1 instance then i got below error when try to see what are the nodes(docker node ls) in the second node and when tried to list the services running(docker service ls).

Error response from daemon: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online

Also no docker process coming up in node 2 which were running in node 1 before stopping the instance. Only the existing process are running. My expectation is after stopping the node1 instance, the procees were running in node 1 has to move to node2. This works fine when a node goes to drain status

1 Answer 1

1

The raft consensus algoritm fails when it cant' find a clear majority.

This means, never run with 2 manager nodes as one node going down leaves the other with 50% - which is not a majority and quorum cannot be reached.

Generally in fact, avoid even numbers, especially when splitting managers between availability zones, as a zone split can leave you with a 50/50 partition - again no majority and no Quorum and a dead swarm.

So, valid numbers of swarm managers to try are generally: 1,3,5,7. Going higher than 7 generally reduces performance and doesn't help availability.

1 should only be used if you are using a 1 or 2 node swarm, and in these cases, loss of the manager node equates to loss of the swarm anyway.

3 managers is really the minimum you should aim for. If you only have 3 nodes, then prefer to use the managers as workers than run 1 manager and 2 workers.

2
  • Can I run my containers in leader node. Or I must run my containers only in worker nodes. Leaving the leader node to only manage. Jun 22, 2021 at 15:55
  • docker, by default, runs service tasks on all nodes. You can add deploy: placement: constraints: in the compose.yml with a clause like node.role==worker or node.role==manager when your swarm is large enough that it would benefit from containers not running on managers unless needed. Jun 22, 2021 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.