Why architecture matters here
The NameNode concentrates three responsibilities that make naive HA impossible: it holds the namespace (every directory, file, and block mapping) in memory for latency; it is the single serialization point for metadata mutations; and it tracks block liveness from thousands of DataNode heartbeats. You cannot just run two of them behind a load balancer — metadata writes must have exactly one writer — and you cannot cold-start a replacement quickly, because loading a 100GB fsimage and replaying edits takes tens of minutes while your entire data platform is down.
The architecture therefore has to answer four questions precisely. Where is the truth? In the edit log, committed to a quorum of JournalNodes — not on any single NameNode's disk. How hot is the spare? The standby replays edits continuously and receives block reports directly from DataNodes, so at failover it needs seconds of catch-up, not a cold load. Who decides who leads? ZooKeeper's ephemeral-node lock, mediated by ZKFC health monitors — an external, quorum-backed arbiter rather than the nodes voting about each other. How is split-brain prevented? Twice: JournalNode epoch numbers make it mathematically impossible for a deposed active to commit edits (writer fencing at the log), and configurable fencing (graceful RPC, then sshfence kill) prevents it from even serving stale reads to clients.
Why study this in 2026, when you may never operate bare HDFS? Because this exact decomposition — replicated write-ahead log, hot standby by log-tailing, external lock service, epoch-based fencing — is the standard recipe. Kafka's controller, Kubernetes' etcd-backed API servers, and most databases' primary election rhyme with it. HDFS just happens to document the recipe unusually well, including the failure drills.
The architecture: every piece explained
Top row: the NameNodes and their shared log. The active NameNode serves every client RPC and writes each namespace mutation to the JournalNodes — typically three or five small daemons — via the Quorum Journal Manager (QJM). A write is acknowledged only when a majority of JournalNodes have durably logged it; with 3 JNs you tolerate one JN failure, with 5 you tolerate two. QJM stamps every writer with an epoch number: when a new active takes over it obtains a higher epoch from the quorum, and JournalNodes thereafter reject writes from any lower epoch — the old active is fenced out of the log by arithmetic, not by hope. The standby NameNode tails committed edit segments from the JNs and applies them to its in-memory namespace, staying seconds behind; it also performs checkpointing (the role the old Secondary NameNode played): periodically merging edits into a fresh fsimage and shipping it to the active, keeping restart-replay bounded. The Observer NameNode — a third role in recent Hadoop — also tails edits and serves read RPCs, with a client-side msync mechanism to bound staleness, offloading the heavy read traffic (listStatus storms from Spark jobs) from the active.
Middle row: election and fencing. Each NameNode host runs a ZKFailoverController (ZKFC) sidecar that health-checks its local NameNode and participates in leader election through ZooKeeper: the active's ZKFC holds an ephemeral lock znode; if the active fails its health check or the ZKFC session dies, the znode vanishes and the standby's ZKFC grabs the lock. Before promoting its NameNode, it executes fencing: a graceful 'cede active state' RPC to the old node, escalating to sshfence (kill the process over SSH) or a custom script (cut power/network) — belt and braces on top of the epoch fencing.
Bottom rows: the data plane's view. DataNodes are configured with all NameNodes and send heartbeats and block reports to each, so the standby's block map is already warm — the piece that makes failover fast, since block locations are never persisted, only reported. Clients use a failover proxy provider: the nameservice URI (hdfs://prod-ns) resolves to the NameNode list, and on a standby exception the client retries the other node with backoff — failover is a retried RPC, invisible to applications above the FileSystem API.
End-to-end flow
Follow a write, then a failover. A client creates /data/events/2026-07-08/part-0001. The RPC lands on the active NameNode (the client learned which node is active on a previous call, or discovers it by trying). The active applies the mutation to its in-memory namespace and writes the edit to the JournalNode quorum; two of three JNs fsync and acknowledge; only then does the client get its lease and DataNode pipeline. The standby, tailing the JNs on its next poll, replays the edit within a second or two. Every hour or so it checkpoints: merges accumulated edits into a new fsimage, uploads it to the active, and both sides truncate old edit segments.
Now the active's motherboard dies at 02:13. Its ZKFC's ZooKeeper session times out (a few seconds); the ephemeral lock znode is deleted. The standby's ZKFC, watching that znode, wins the election and begins promotion. Fencing first: the graceful RPC to the dead host times out, sshfence fails too (the machine is off), but that is acceptable — QJM epoch fencing already guarantees the old active cannot commit anything, and the operator-configured fencing policy allows proceeding when the node is provably unreachable. The standby requests a new epoch from the JournalNodes, replays the final unapplied edit segments (it was seconds behind), and declares itself active. Total: typically 30–120 seconds from failure to serving writes.
Client experience during the window: an application calling fs.create() gets a retriable exception, the failover proxy provider flips to the other NameNode, sleeps briefly, retries, and succeeds. Long-running jobs see a latency blip, not a failure. DataNodes never blinked — they were already heartbeating to both nodes, so the new active has a complete block map and immediately starts approving replication work. When the dead host is repaired, its NameNode restarts as standby, bootstraps from the latest fsimage, tails the journal, and its ZKFC rejoins the election as the new backup. The 02:13 page becomes a morning ticket instead of an outage bridge.