Why architecture matters here
Leader election failures are the classic "two leaders wrote at once" incident. Without fencing tokens, a stale leader can accept writes after being fenced out. Without downstream verification, the fencing token is decorative.
The architecture matters because election is only half the job — enforcing single-leader semantics on downstream resources is the other half.
With the pieces mapped, leader election becomes a reliable primitive.
The architecture: every piece explained
The top strip is the election. Candidates race to acquire leadership. Coordination service — etcd, ZK, Consul — arbitrates. Lease + TTL holds leadership for a term. Fencing token is a monotonically increasing epoch that resources use to reject stale writes.
The middle row is failure handling. Heartbeat renews the lease. Failure detection is the TTL expiry when heartbeats stop. Failover lets a new candidate acquire lease + fresh fencing token. Split brain check: even if the old leader thinks it still has the lease, downstream resources reject its stale fencing token.
The lower rows are ops. Downstream verify: every write includes the fencing token; resources track the highest seen and reject lower. Observability tracks leadership changes and durations. Ops handles election storms with jitter and drills failover.
End-to-end flow
End-to-end: three schedulers race to be leader. Node A wins, gets lease + fencing token=42. A writes to state store; store records fencing=42. A's network partitions; heartbeats miss. Lease expires. Node B acquires; new fencing=43. B writes; store accepts. A recovers, thinks it's still leader, writes with fencing=42; store rejects because 42 < 43. A detects mismatch, steps down. No corruption. Observability shows the leadership handoff time within SLA.