Why architecture matters here
Consensus is expensive. Every write requires a round-trip to a majority of replicas plus log persistence. Distributing decisions is the point, but distributing everything is the anti-pattern. Consensus is right for metadata, leader election, and small critical state — not for user data at scale.
The architecture matters because incidents almost never come from bugs in the algorithm itself — Raft is well-studied. Incidents come from operational surprises: election storms when heartbeats miss, log growth when snapshots do not compact, split brain when membership changes are misused, or apply lag when the state machine is slow.
Knowing the pieces lets you diagnose those incidents quickly and, more importantly, avoid them by picking sizing and workload correctly.
The architecture: every piece explained
The top strip is the role machine. Follower is the default: passively receives log entries from the leader. Candidate is what a follower becomes if it stops hearing heartbeats before its Election timer fires; the candidate requests votes from peers. Leader is elected on quorum agreement; it accepts writes, appends to its log, and sends heartbeats + entries to followers.
The middle row is the log machine. Log entries carry (term, index, command). The Commit index advances when the leader sees a majority of followers acknowledge an index; once committed, the entry is applied to the State machine. When the log grows large, a Snapshot captures the state machine at an index; earlier log entries can be discarded and new followers can install the snapshot instead of replaying from zero.
The lower rows are the operator surface. Membership changes use joint consensus to add or remove a replica safely without risking split brain. Client interface proxies reads and writes to the leader; some Raft variants add read-index or lease-based reads for lower-latency linearizable reads. Observability watches term (should rarely change), commit lag between followers, election frequency (a healthy cluster has one election per leader lifetime), log size, and apply latency.
End-to-end flow
End-to-end: a client writes a config value. The API server proxies to the leader. The leader appends a log entry at index N with the current term T. It replicates to two followers. Both persist and acknowledge. The leader advances its commit index to N and applies the entry — the config now reflects the change. Followers observe the commit index in the next heartbeat and apply on their side. The state machine is linearizable across the cluster. Later a network partition isolates the leader; heartbeats stop; followers' election timers fire; one becomes candidate, gathers votes, wins, and becomes leader with term T+1. The old leader eventually sees the higher term and steps down. No committed entries are lost because the new leader is elected only if its log is at least as up to date as the majority.