Why architecture matters here
Consensus matters because everything durable in a distributed system rests on it. Metadata service? Consensus. Leader election? Consensus. Distributed transactions? Consensus at commit. Cluster membership? Consensus. Configuration store? Consensus. Without it, you can build eventually consistent systems that reconcile over time — but you cannot build systems where "committed" means "will be observed by every future read."
The classic FLP impossibility result says asynchronous consensus is impossible if messages can be arbitrarily delayed. Real systems escape by adding timing assumptions (leaders time out, followers restart, network heals). The trick is to make those assumptions explicit and to design the algorithm so that even when they are violated (network partition, clock skew, GC pause), the system remains safe — it may pause and refuse to make progress, but it never violates consistency.
Ops matters here more than most places. A consensus cluster with a broken quorum is not degraded; it is dead. Adding, removing, and replacing nodes is a delicate operation. Snapshots, log compaction, and disk failures each have their own gotchas. Teams that operate consensus well practice these operations regularly; teams that don't discover the gaps at 3 AM.
The architecture: every piece explained
Walk the diagram. Every consensus system has these pieces in some form.
Client. The outside world. Clients issue read and write requests. Reads may be served from any replica if you accept staleness; linearizable reads must go through the leader (or use special quorum-read protocols). Writes always go through the leader.
Coordinator. Sometimes an SDK function, sometimes a proxy, sometimes a service mesh. It routes client requests to the current leader. When leadership changes, it re-discovers via a directory service or by probing.
Leader. The current leader is the only node that appends new entries to the log. Leadership is time-bounded ("term" in Raft) and re-elected when the current leader fails or partitions off. Followers only accept AppendEntries from the current-term leader; older-term messages are rejected. This term system is the single elegant idea that keeps consensus safe under partition healing.
Replicated Log. An append-only sequence of entries, each with an index and a term. The log is the ground truth. When a majority of nodes have durably stored an entry, it is "committed" and will be applied to every state machine in order. Consensus is fundamentally about agreeing on the log; the state machines are downstream from that.
Followers. Every non-leader node. Followers accept AppendEntries messages, verify the previous entry matches (log matching property), append, and acknowledge. The leader tracks each follower's matchIndex (highest replicated) and nextIndex (next to send). When a follower falls behind, the leader backs up and re-sends.
State Machines. Each node maintains an identical state machine that applies committed log entries in order. Because every node applies the same sequence to the same starting state, every node arrives at the same final state. This is the core reason consensus provides linearizability: an outside observer cannot distinguish which node they are talking to.
Snapshot Store. Logs cannot grow forever. Periodic snapshots capture the state machine at a specific log index, letting older log entries be compacted. When a lagging follower needs to catch up beyond the current log start, the leader ships the snapshot first, then any log entries after it.
End-to-end write and recovery flow
Follow a write. The client sends "SET foo=bar" to the coordinator. The coordinator routes it to the current leader. The leader appends the command as a new log entry at index N, term T, and sends AppendEntries to all followers with that entry.
Each follower checks that its log at index N-1 matches term T-1 (the log-matching property). If it does, the follower appends the entry to durable storage and returns acknowledgment. If it doesn't, the follower rejects; the leader decrements nextIndex for that follower and re-sends earlier entries until they match.
Once the leader sees acknowledgments from a majority (including itself), it marks entry N as committed. It piggybacks the commit index on the next AppendEntries so followers know to advance their commit index too. Each node applies the committed entry to its state machine; the leader responds to the client with success.
Leader failure. The leader crashes. Followers time out on missed heartbeats. One follower increments its term and becomes a candidate; it requests votes from the others. A majority grants their votes, and the candidate becomes the new leader for the new term. The new leader may need to send earlier log entries to catch up followers who missed them; log matching ensures no committed entry is lost.
Snapshot. The state machine grows past a threshold. The leader takes a snapshot, records the snapshot's log index and term, and compacts entries up to that index. Followers that were up-to-date do the same. A lagging follower that needs entries older than the snapshot receives the snapshot via InstallSnapshot RPC and then catches up on newer entries.