To understand etcd, you need to be familiar with three key concepts relating to storage management and clusters:
In Raft-based systems, the cluster elects a leader for a given term. The leader processes all the storage requests that require the consensus of the cluster. Requests that do not require cluster consensus (reads, for example) can be answered by any member of the cluster. If the leader accepts a change, etcd makes sure it replicates the information to the follower nodes. Once the followers have confirmed receipt, the leader commits the changes.
This kind of system, where a leader uses an etcd database to coordinate changes with the nodes in the cluster, is very valuable in distributed applications. If changes would affect how a node operates, the node can block them. This ensures that the application remains stable and minimizes related problems.
If a leader dies or fails to respond within a certain time, the remaining nodes in the cluster elect a new leader. Each node has a random timeout setting that determines how long it waits until calling for a new election and declaring itself as a candidate. These timeout settings are controlled by a special timer within each node and are designed to allow a new node to become the leader as quickly as possible.
To ensure that there is always a majority in an election, there must be an odd number of nodes in the cluster. For performance reasons, clusters should not have more than seven nodes.