Why architecture matters here
Leader election matters because a system without a stable leader is a system without a coordinator. Every write goes through the leader; every membership change goes through the leader; every transaction commit goes through the leader. Frequent elections mean minutes of unavailability per week. Rare elections that go wrong mean split-brain and data loss. The election protocol is the boundary between a well-behaved cluster and a flakey one.
The specific failure modes are what make leader election worth studying. Network partitions can create ghost leaders that briefly accept writes. Slow GC pauses can trigger unnecessary elections. Asymmetric partitions (leader can send but not receive) look normal from one side and dead from another. Term-based election avoids most of these by ensuring only one leader per term and requiring quorum for progress; but the details of how you avoid spurious elections and how you resolve split votes are what separates a good implementation from a fragile one.
Operationally, leader election is where you feel the pain of clock skew, disk stalls, and slow networks. Any anomaly in the environment expresses itself as an election. Tuning heartbeat intervals, election timeouts, and PreVote thresholds is one of the most impactful things an SRE can do; misconfigured timeouts cause flapping that looks like an application bug.
The architecture: every phase explained
Walk the diagram left to right, top to bottom.
Steady state. The leader sends AppendEntries heartbeats — either empty or with new log entries — to every follower every ~50 milliseconds. Followers reset their election timers on each heartbeat. As long as heartbeats flow, no election happens. This is 99%+ of the cluster's time.
Missed heartbeat. A follower's election timer expires. Timers are randomized — say 150-300 ms — so different followers time out at different moments; the first one to time out has a head start on becoming the next leader. Randomization is what prevents synchronized election storms.
Candidate. The timed-out follower transitions to candidate. It increments its term (the monotonic election counter), votes for itself, and starts requesting votes. In implementations without PreVote, this is where a partitioned follower keeps bumping the term unnecessarily; PreVote avoids that.
PreVote (optional but recommended). Before actually bumping its term, the candidate polls followers: "if I ran, would you vote for me?" If not enough would (say, because the current leader is still healthy), the candidate returns to follower without touching its term. This subtle change eliminates a common bug where a partitioned node repeatedly bumps its term and disrupts the cluster when it heals.
RequestVote RPC. The candidate sends a RequestVote message to every follower. The message includes the candidate's new term, its lastLogIndex, and its lastLogTerm. Followers use these to decide whether the candidate has a sufficiently up-to-date log to become leader.
Followers vote. Each follower grants a vote if: (1) it has not voted for anyone else in this term, (2) the candidate's log is at least as up-to-date as its own, and (3) the term is higher than or equal to its current term. Otherwise it rejects. The "up-to-date log" check is what preserves the safety property that no committed entry can be lost during election.
Majority reached. If the candidate collects votes from a strict majority of the cluster (including itself), it becomes leader for the new term. It immediately sends empty AppendEntries heartbeats to establish authority; this prevents followers from timing out and starting their own elections.
Split vote. If two candidates each fail to reach majority, both time out and retry with randomized backoff. Because the backoff is random, one of them will start first next time and likely win. Split votes are rare in practice with well-tuned timeouts.
End-to-end election trace
Trace a real election. The leader's node has a 3-second GC pause. Followers stop receiving heartbeats. The first follower's election timer expires at 200 ms.
If PreVote is enabled, that follower polls the others: "would you vote for me if I ran?" If the previous leader is genuinely down or unreachable, the majority responds yes, and the candidate proceeds to a real election. If the previous leader is alive (just paused), most followers still have valid leader leases and respond no, and the would-be candidate goes back to being a follower without bumping the term.
Assume the leader is really down. The candidate bumps its term from 42 to 43, votes for itself, and sends RequestVote to the other two followers. Each checks: is my term ≤ 43? Yes. Have I already voted in term 43? No. Is the candidate's log at least as up-to-date as mine? Yes (it has the same last index and term). Vote granted.
The candidate collects two of two remaining votes, plus its own — three of four total. Majority. It transitions to leader for term 43 and immediately sends heartbeats. Followers accept the heartbeats and reset their election timers. Normal operation resumes.
Meanwhile the old leader recovers from its GC pause. It tries to send heartbeats as term 42. Followers reject because they now know about term 43. The old leader sees a higher term in the reject responses, updates its own term to 43, and steps down to follower. The cluster is coherent again.
Total unavailability: from the first missed heartbeat to the first successful new heartbeat, typically 300-500 milliseconds. Well-tuned clusters make this invisible to users.