6

Lets say we have a test setup of 10 nodes, 4 managers and 6 workers.

When the leader manager fails, the other 3 managers will chose another manager as leader.

When this leader as well fails, we only have 2 managers left out of 4. The other managers then say

Error response from daemon: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online.

Because we have not more than half of the managers left, they will not be able to chose a new leader although 2 managers of the cluster are left.

My question is

  1. the sense of this rule, because the cluster is without a leader and not manageable anymore as long as no additional managers are added to the cluster, although there are 2 managers available.
  2. Why should I chose the role worker for nodes at all? What advantage are there to have nodes as workers? Managers also act as workers by default only with the disadvantage that they cannot take over when manager nodes fail.
2

1 Answer 1

4
  1. Docker recommends to use a system with odd number of manager nodes. So your initial setup of 4 manager is as good as having 3 manager nodes. It is recommended that you start with 5 nodes, as you are loosing 2 nodes. Also, isn't there any serious issue to be addressed in the way you are using? (loosing so many nodes is not a good sign)

If the swarm loses the quorum of managers, the swarm cannot perform management tasks. If your swarm has multiple managers, always have more than two. To maintain quorum, a majority of managers must be available. An odd number of managers is recommended, because the next even number does not make the quorum easier to keep. For instance, whether you have 3 or 4 managers, you can still only lose 1 manager and maintain the quorum. If you have 5 or 6 managers, you can still only lose two.

  1. Having a dedicated worker nodes makes sure that they won't participate in the Raft distributed state, make scheduling decisions, or serve the swarm mode HTTP API. So the complete compute power of these nodes are dedicated specifically to run the containers.

because manager nodes use the Raft consensus algorithm to replicate data in a consistent way, they are sensitive to resource starvation

The quotes are taken from the docker official documentation link

2
  • So the difference having nodes as workers and not all nodes as managers is only about the computing and networking resources that would be also used for Raft, etc.? Are these really significant or aren't they neglectable in order to have more stability in the cluster? Of course my example from above with 4 managers and 2 managers failing (so 50%) was only an example, but the thing about high availability is especially about special cases. See e.g., Amazon S3 disruption in Feb 2017. So I guess for many services having all nodes as managers is worth the bit overhead of raft consensus, etc.
    – Cravid
    Apr 4, 2018 at 12:59
  • 1
    See there are always high chances of nodes going down with high compute operations => nodes running the containers. So having all nodes as manager only, you risk loosing more nodes, if at all your container drains too much of resources. Also having a few dedicated nodes as manager, makes sure that they stay up and manage the worker nodes efficiently. Note: Docker did mentioned not go beyond 7 manager nodes for the very same reason.
    – Vamsi
    Apr 5, 2018 at 5:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.