HDFS High Availability — Active/Standby NameNodes, JournalNodes, ZooKeeper Failover

Why it matters

Enterprise Hadoop clusters run twenty-four-seven and cannot tolerate NameNode downtime for reboot, patching, or hardware failure. A cluster that goes read-only for ten minutes during a planned NameNode restart is fine for a research lab but unacceptable for a production data platform that feeds real-time dashboards and analytical pipelines. HA removes this friction and lets you run rolling upgrades of the NameNode without a maintenance window.

HA is also the difference between a cluster that survives a datacenter power event and one that requires a manual recovery procedure taking hours. Automatic failover to a standby NameNode is measured in tens of seconds; manual recovery of a crashed NameNode is measured in minutes to hours depending on how large the edit log has grown since the last checkpoint.

The architecture

An HA deployment has two NameNode processes running on separate hosts. Exactly one is active at any moment and serves all client RPCs. The other is standby and does not serve reads or writes; it only maintains a warm in-memory copy of the namespace by continuously tailing the edit log.

The edit log is stored on a JournalNode quorum, typically three or five JournalNode processes on three or five separate hosts. Both NameNodes read and write the log through the quorum. Writes must be acknowledged by a majority of JournalNodes before they are considered durable, which is how you get fault tolerance without split brain.

How it works end to end

Both NameNodes run alongside a small process called the ZooKeeper Failover Controller, or ZKFC. The ZKFC on each NameNode maintains a ZooKeeper session and races to acquire a shared lock znode. Whoever holds the lock is the active. When the active NameNode dies or its ZKFC loses its ZooKeeper session, the lock is released. The standby's ZKFC immediately detects this and grabs the lock, but before it lets the standby NameNode promote, it fences the previous active. Fencing means using a defined mechanism, usually sshfence, to forcibly kill the old active's NameNode process so there is no possibility of two active NameNodes writing to the JournalNodes at once.

Once fencing succeeds, the ZKFC signals the standby NameNode to become active. It replays any final edits from the JournalNode quorum, opens itself for client RPCs, and the cluster is back in service. Clients discover the new active through the logical service name that resolves to whichever NameNode currently owns the ZooKeeper lock.

The whole failover takes twenty to sixty seconds in typical setups. Most of that time is ZooKeeper session timeout detection, which by default is fifteen seconds. Aggressive tuning can drive failover under fifteen seconds total but risks false failovers under network jitter, so most sites leave the defaults.

Quorum Journal Manager - where the truth lives

What has to be replicated is not the namespace but the edit log. Every metadata mutation is appended to it before the active NameNode acknowledges the client, so any machine that can replay the log can rebuild the namespace. HA therefore reduces to one problem: storing a write-ahead log that loses no committed edit and admits no second writer.

The Quorum Journal Manager solves it. The active writes edit batches to a set of JournalNode daemons - three or five, on separate hosts - named by a qjournal:// URI in dfs.namenode.shared.edits.dir. A batch commits only when a majority of JournalNodes have fsynced it: 3 JNs survive 1 failure, 5 survive 2, and 4 is pointless because 3 of 4 is still the majority.

JournalNodes are deliberately tiny - edit segments under dfs.journalnode.edits.dir, no fsimage, no block map - and normally sit on existing master hosts. Their one hard requirement is a fast fsync; a JN on a contended disk adds latency to every namespace write in the cluster. The older alternative, a shared NFS filer, moves the single point of failure into the filer and demands genuine STONITH fencing, since NFS has no concept of a single writer.

Epoch numbers and fencing - two defences against split brain

A quorum log alone does not stop two NameNodes both believing they are active. Epoch numbers do. A NameNode transitioning to active first runs a recovery round: it proposes an epoch higher than any it can see, and each JournalNode that accepts promises never again to honour a request carrying a lower epoch. Once a majority has promised, the new active recovers the last in-progress segment - picking the longest version a majority agrees on and forcing the rest to match - and only then begins appending.

The consequence is decisive. A deposed active stuck in a ninety-second GC pause, or partitioned and unaware it lost the lock, still holds the old epoch: every JournalNode rejects its next write, it can never reach a majority, and it aborts. It is fenced out of the log by comparing two integers rather than by anyone reaching it - which matters, since the failure that triggered failover often also makes the old host unreachable.

Epoch fencing stops writes, not reads: a stale active can still serve listings minutes out of date, which is why an explicit fencing step exists. Before promoting, the failover controller runs the methods in dfs.ha.fencing.methods in order until one succeeds. sshfence logs into the old host with the key from dfs.ha.fencing.ssh.private-key-files and kills the NameNode process, bounded by dfs.ha.fencing.ssh.connect-timeout. shell(...) runs any command and treats exit 0 as success - the hook for IPMI or switched-PDU STONITH, the only fencing that is honest when a machine is wedged rather than dead. Since QJM already blocks the old active's writes, some deployments append shell(/bin/true) so failover never stalls; that is defensible only with a real power fence ahead of it.

ZKFC, hot standby, and checkpointing

Health monitoring and election

Each NameNode host runs a ZKFailoverController beside it, and it calls monitorHealth on its local NameNode on a short interval. A NameNode that is up but sick - a failed shared-edits write, a full edit directory - reports unhealthy, and its ZKFC gives up the lock rather than waiting for the process to die. Election runs through an ephemeral znode under /hadoop-ha/<nameservice>: whoever holds ActiveStandbyElectorLock is active, and it evaporates when the ZooKeeper session expires. A sibling persistent znode, ActiveBreadCrumb, records who was last active so the incoming ZKFC knows which host to fence. Detection latency is thus ha.zookeeper.session-timeout.ms.

Why the standby is hot

Block locations are never persisted in the fsimage; they are rebuilt from DataNode block reports. A standby that learned them only after promotion would sit waiting on thousands of DataNodes, so every DataNode registers with, heartbeats to, and block-reports to both NameNodes. The standby's block map stays warm, promotion needs no block-report wait, and its heap must be sized identically to the active's. See HDFS DataNode and blocks and replication.

Checkpointing, which the standby now owns

The standby also does the Secondary NameNode's old job: on dfs.namenode.checkpoint.period or dfs.namenode.checkpoint.txns it merges edits into a fresh fsimage and uploads it to the active. You do not run a Secondary NameNode in an HA cluster; configuring one anyway is a classic setup error. Detail: NameNode checkpointing.

A working HA configuration

The nameservice ID is the logical name clients use; the NameNode IDs label the two processes.

hdfs-site.xml   (nn2 address pairs elided)
  dfs.nameservices                     = prodns
  dfs.ha.namenodes.prodns              = nn1,nn2
  dfs.namenode.rpc-address.prodns.nn1  = nn-a:8020
  dfs.namenode.shared.edits.dir        = qjournal://jn1:8485;jn2:8485;jn3:8485/prodns
  dfs.journalnode.edits.dir            = /data/jn
  dfs.ha.automatic-failover.enabled    = true
  dfs.ha.fencing.methods               = sshfence
                                         shell(/usr/local/bin/power-fence.sh)
  dfs.ha.fencing.ssh.private-key-files = /var/lib/hdfs/.ssh/id_rsa
  dfs.client.failover.proxy.provider.prodns =
      org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

core-site.xml
  fs.defaultFS                         = hdfs://prodns
  ha.zookeeper.quorum                  = zk1:2181,zk2:2181,zk3:2181

Bring-up order matters - JournalNodes first, then seed nn-b from nn-a:

hdfs --daemon start journalnode     # all JN hosts
hdfs namenode -bootstrapStandby     # nn-b: pulls fsimage from nn-a
hdfs zkfc -formatZK                 # once: creates /hadoop-ha
hdfs --daemon start zkfc            # both NN hosts

hdfs haadmin -getServiceState nn1
hdfs haadmin -failover nn1 nn2      # graceful

haadmin -failover asks the active to cede, fences it, promotes the peer. Reach for -transitionToActive --forcemanual only with automatic failover disabled and proof the other node is dead: it bypasses the elector, and is the fastest way to build a split brain by hand.

Client failover and observer reads

Applications address the cluster as hdfs://prodns/..., never as a hostname, and discovery is by trial: an RPC landing on the standby is not redirected - the server throws StandbyException, the client's retry handler treats that as "wrong node, not an error", flips to the other address, and retries with exponential backoff bounded by dfs.client.failover.max.attempts. That loop absorbs the outage window. Mutating calls stay safe via a NameNode-side retry cache keyed by client and call ID, whose state rides in the edit log and survives promotion; a long job sees a latency spike, not a failure.

Read-heavy clusters hit an RPC ceiling on the active - listStatus storms from query planners - long before any storage limit, while the standby idles. An Observer NameNode tails the log like a standby but also serves reads. Two mechanisms keep that safe: observers tail in-progress edit segments (dfs.ha.tail-edits.in-progress), pulling lag toward the sub-second range, and an alignment context has each client carry the highest transaction ID it has seen, so an observer behind that point waits rather than serving a read that goes backwards in time. A client needing to see its own recent write calls msync(). Observers are created with haadmin -transitionToObserver, opted into via ObserverReadProxyProvider, and take no part in the election - a scaling feature, not an availability one.

Failover time and what dominates it

An unplanned failover is a sum of four terms, and only one is usually large. Detection is the ZooKeeper session timeout - the ZKFC's session must actually expire before the lock znode disappears - and it normally dominates. Fencing costs whatever your methods cost: a cede RPC to a dead host burns its full timeout, and sshfence to a powered-off machine burns the SSH connect timeout on top. Log recovery - new epoch, segment recovery, replaying the few untailed transactions - is a second or two. Warm-up is near zero for the block map, though the new active's RPC queue spikes as every client and DataNode arrives at once.

Planned failover skips detection entirely and finishes in seconds, so patch NameNodes with a deliberate haadmin -failover rather than killing the active and letting HA notice. Resist shrinking the session timeout aggressively: a large-heap NameNode can pause for a multi-second GC, and a timeout below that pause turns a survivable stall into a real failover. A hanging fence is the other common way a failover takes minutes - test that path by powering off the active and timing the transition.

Operational failure modes

JournalNode quorum loss. An active that cannot write to a majority of JNs does not degrade - it aborts, because continuing would mean accepting edits no quorum has logged. Two dead JNs out of three take the cluster down even though both NameNodes are healthy, so restart JournalNodes one at a time and never in parallel.

A slow JournalNode disk. Write latency is bounded by the slowest JN in the committing majority, so a JN sharing a spindle with a busy log volume surfaces as unexplained latency on every create. Treat JN fsync latency as a first-class metric.

Both NameNodes standby. Almost always ZKFC: not started, zkfc -formatZK never run, dfs.ha.automatic-failover.enabled left false, or the ensemble unreachable. A ZooKeeper outage does not demote a running active - it removes only the ability to fail over.

Standby falling behind. Compare LastAppliedOrWrittenTxId on both NameNodes' JMX endpoints; a growing gap means slow promotion and usually an edit-tailing or JN problem. A standby down long enough for old edit segments to be purged needs bootstrapStandby again, not a restart.

What HA does not solve

HA buys availability of the namespace service, and nothing else. It does not scale the namespace - both NameNodes hold the identical namespace in the identical heap, so the file and block ceiling is unchanged; that is a job for federation or router-based federation. It does not protect against logical damage, since a mistaken recursive delete replicates within a second - that is what snapshots are for. Nor does it substitute for knowing what safe mode waits on.

The cost is one extra NameNode-class host, three colocatable JournalNode processes, and the discipline to test failover on a schedule - untested HA is not HA. For this machinery from an architectural angle see NameNode HA architecture; for the master itself, HDFS NameNode.