Why architecture matters here
Architecture matters because the naive approaches are all subtly broken. 'Whoever holds the database row is leader' fails when the holder's connection drops but its process lives on, still believing it holds the row. 'Whoever has the lowest node ID' fails the moment the network partitions and each side sees a different lowest ID. 'A shared lock with a timeout' fails when the lock expires because the leader was paused, a second node grabs it, and then the first node resumes mid-operation — now two nodes genuinely hold the lock at overlapping times. Every one of these produces split-brain under conditions that will occur in production, not in theory.
The correct architecture separates two guarantees that beginners conflate. Liveness: eventually exactly one leader exists and can make progress. Safety: at no time do two leaders both perform leader actions that must be serialized. You cannot have perfect liveness and safety together with unreliable networks (that is the practical face of FLP and CAP), so real systems choose safety and accept that during a partition the minority side simply cannot lead. A minority that cannot reach quorum must step down rather than assume it is still leader — even though, from inside that partition, it cannot tell a network cut from everyone else dying.
Getting this right is what lets higher layers stay simple. A shard writer, a job scheduler, or a primary database can be written as 'if I am leader, do X' precisely because the election layer guarantees the 'am I leader' answer is safe. Get election wrong and every downstream component inherits split-brain bugs that are almost impossible to reproduce or debug.
These bugs are uniquely nasty because they are rare, non-deterministic, and destructive. A split-brain window might open once a month, for a few seconds, only when a GC pause coincides with a network hiccup — impossible to reproduce on demand, invisible in normal testing, and yet capable of double-charging a customer or corrupting a shard in that brief window. By the time you notice the data damage, the triggering conditions are long gone and the logs show two nodes that each believed, correctly from their own vantage point, that they were leader. This is why the safety machinery — quorum and fencing — must be designed in from the start rather than bolted on after the first incident: you cannot debug your way to correctness here, you can only architect it.
The architecture: every piece explained
Candidates are the nodes eligible to lead. To become leader, a candidate must win agreement from a quorum — a strict majority of a fixed membership. Majority quorums are the trick that prevents two leaders: any two majorities of the same set must overlap in at least one node, and that node will not vote for two leaders in the same term, so two simultaneous majorities for different leaders are impossible. This is why consensus systems (Raft, Multi-Paxos) and coordination services (ZooKeeper, etcd) size clusters as 3, 5, or 7: an odd count maximizes fault tolerance per node while keeping a clear majority.
Every leadership is stamped with a term (Raft) or epoch (ZooKeeper zxid, etcd revision): a monotonically increasing integer that increments on each election. The term is the backbone of correctness — it lets everyone totally order leadership periods and reject anything from an older term. The current leader proves it is alive by sending heartbeats; it holds a lease with a TTL, and if it fails to renew before the TTL expires, followers conclude the leader is gone and start a new election with a higher term. The TTL directly sets the failover budget: short TTLs fail over fast but risk falsely evicting a briefly-slow leader; long TTLs are stable but slow to recover.
The final, essential piece is the fencing token: the current term number, handed to the leader and attached to every action it performs against shared resources (a storage service, a database, a lock manager). Downstream resources remember the highest token they have seen and reject any request carrying a lower one. So if a paused old leader wakes up in term 5 while term 6 has already begun, its writes carry token 5, the storage layer has already seen token 6, and the stale writes are refused. Fencing is what turns 'we try to have one leader' into 'a second leader cannot do damage even if it exists for a moment.'
It helps to separate the two families of implementation you will encounter. In a consensus-based system (Raft, Multi-Paxos), leadership and the replicated log are one mechanism: winning an election in term T also makes you the authority on the log, and the same quorum that elected you commits your writes, so leadership and data consistency are inseparable. In a coordination-service approach (ZooKeeper ephemeral nodes, etcd leases, Consul sessions), the election is delegated to an external, already-consistent store, and your application merely holds a lease it must renew; the store handles quorum internally, and you get a simple 'am I leader' boolean plus a fencing number. The consensus approach is heavier but keeps everything in one system; the coordination approach is lighter and reuses a store you likely already run, at the cost of a hard dependency on that store's availability.
End-to-end flow
Start with a healthy cluster of five nodes. Node C is leader in term 7, holding a lease with a 6-second TTL and heartbeating every 2 seconds. Followers reset their election timers on each heartbeat. All writes to the shared shard go through C, and each write to the storage layer carries fencing token 7. Clients that contact a follower are redirected to C. Life is calm.
Now C's host suffers a stop-the-world GC pause of 8 seconds. Heartbeats stop. After the 6-second TTL, followers' election timers fire. Node A, timing out first, increments the term to 8 and campaigns: it requests votes from the others. B, D, and E — having also missed heartbeats — grant their votes for term 8. A now has 4 of 5 (a quorum), declares itself leader in term 8, obtains fencing token 8, and begins heartbeating. Failover completed in roughly one TTL plus one election round.
Then C's GC finishes and it resumes execution, still believing it is leader in term 7. It tries to write to the shard with token 7. But the storage layer has already accepted a write from A carrying token 8, so it has advanced its highest-seen token to 8 and rejects C's token-7 write. C also discovers, on its next heartbeat attempt, a reply carrying term 8, learns it has been superseded, and immediately steps down to follower. At no point did two leaders successfully mutate shared state: A led, C's stale writes were fenced out, and C rejoined cleanly. This is the flow every leader-election design must produce — bounded failover for liveness, fencing tokens for safety.
Contrast this with what happens if a partition instead splits the cluster into a 2-node and a 3-node group while C is healthy. C, on whichever side it lands, can only remain leader if it can still heartbeat to a quorum. On the 3-node side it can; on the 2-node side it cannot renew its lease against a majority, so it steps down and stops issuing writes — even though from inside that partition everything looks fine and C cannot distinguish a network cut from the other three nodes dying. The 3-node side keeps C (if C is there) or elects a new leader in a higher term. When the partition heals, the minority nodes discover the higher term, discard any local uncommitted state, and rejoin as followers. No write was ever accepted by two leaders, because acceptance required a quorum and only one side could ever have one — the overlap property of majorities doing its job.