Why the number of Master node should always be odd. 3,5 or 7…?
The idea is to avoid split-brain scenarios if masters go down.
One master is all that is technically needed. However, if that goes down, the cluster is down in relation to updates etc.
However, 2 nodes isn’t any help, as if the there is a network partition (where the two master nodes can’t communicate, but both are still alive and working) neither of the two remaining nodes knows if the other node is still working. This is a form of deadlock – one cannt update the cluster in case the other is still alive and making a different update.
3 helps because if there is a partition – either side of the partition will have the upper hand (two masters) and the other side of the partition will only have master. In this scenario, both side know if they have a majority or not. Same goes for 5 and 7.
More than 7 can introduce latency when it comes to achieving quorum etc. In simple terms, think of it like deciding where to eat – it’s relatively easy for 4 or 5 people to decide. It’s probably harder if you’ve got 20 people.
as @nigelpoulton explained, to give one bit more detail, the one component in the masters that dictates this high availability/scalability requirement is etcd
Just like most other distributed systems, the leader election here is happening via Raft Consensus algorithm. You can search more on this topic, in case interested to know in deep.