Why the number of Master node should always be odd. 3,5 or 7…?
3 Answers
Hi kaushroa.
The idea is to avoid split-brain scenarios if masters go down.
One master is all that is technically needed. However, if that goes down, the cluster is down in relation to updates etc.
However, 2 nodes isn’t any help, as if the there is a network partition (where the two master nodes can’t communicate, but both are still alive and working) neither of the two remaining nodes knows if the other node is still working. This is a form of deadlock – one cannt update the cluster in case the other is still alive and making a different update.
3 helps because if there is a partition – either side of the partition will have the upper hand (two masters) and the other side of the partition will only have master. In this scenario, both side know if they have a majority or not. Same goes for 5 and 7.
More than 7 can introduce latency when it comes to achieving quorum etc. In simple terms, think of it like deciding where to eat – it’s relatively easy for 4 or 5 people to decide. It’s probably harder if you’ve got 20 people.
HTH
One additional thing to consider is that the more nodes you have in a cluster, the more likely you are to run into network partitions. I ran into this where we thought at first we’d have a more resilient cluster but we ended up just getting more splits! This was with elastic several years ago but clusters tend to have similar behavior. Best stability was 5 or 7 nodes for quorum.
Thank you Nigel Poulton!!
as @nigelpoulton explained, to give one bit more detail, the one component in the masters that dictates this high availability/scalability requirement is etcd
Just like most other distributed systems, the leader election here is happening via Raft Consensus algorithm. You can search more on this topic, in case interested to know in deep.
Let the node be 3