HDFS Checkpoints and JournalNodes — How the Namespace Stays Durable

Transaction IDs — the number that makes two files one history

The NameNode assigns every namespace mutation a monotonically increasing 64-bit transaction ID. Creating a file, allocating a block, renaming a directory, changing a replication factor, granting a lease — each becomes one or more numbered records. That single counter is the reason the durability design works at all: it turns two unrelated-looking artifacts into a single ordered history.

An fsimage is not a backup in the usual sense. It is a serialization of the entire namespace as of a specific transaction ID, and it carries that ID in its own filename. An edit log segment likewise declares the ID range it covers. So the current namespace has an exact definition rather than a fuzzy one: load the newest fsimage, then replay every edit record whose ID is greater than the image's ID, in order, with no gaps. If a record is missing from the middle of that sequence, startup does not silently continue — it fails, because a namespace assembled from a punctured history would be quietly wrong rather than loudly absent.

This is also what makes the two NameNodes in an HA pair comparable. Both expose their position as a transaction ID, so "is the standby caught up?" is a subtraction rather than a guess, and "does this image match this journal?" is an integer comparison rather than a checksum of the world. Almost every operational question in this article reduces to reading one of those numbers.

What is actually on disk

Point dfs.namenode.name.dir at a directory and the NameNode will populate a current/ subdirectory inside it. What lands there is legible if you know the naming scheme:

/data/nn/current/
  VERSION                                   # namespaceID, clusterID, blockpoolID, layoutVersion, cTime
  seen_txid                                 # highest txid this node must be able to reach
  fsimage_0000000000000098765               # namespace snapshot as of txid 98765
  fsimage_0000000000000098765.md5           # its checksum, verified on load and on transfer
  edits_0000000000000098766-0000000000000099412   # a finalized segment
  edits_inprogress_0000000000000099413      # the segment currently being appended to

VERSION is the identity file. Its clusterID and blockpoolID are what stop a NameNode from adopting storage that belongs to a different cluster, and its layoutVersion is what an upgrade changes. seen_txid is the guard against silent rollback: it records how far the node knows the history has advanced, and a NameNode that finds on-disk artifacts unable to reach that point refuses to start rather than serving an older namespace.

Both dfs.namenode.name.dir and dfs.namenode.edits.dir accept a comma-separated list, and the NameNode writes to all of them — the standard local redundancy for the image. In an HA cluster the authoritative edit stream moves out from under this directory entirely and is addressed by dfs.namenode.shared.edits.dir with a qjournal:// URI, though local copies are still maintained. Setting dfs.namenode.name.dir.restore lets the NameNode re-attach a storage directory that failed and later came back, instead of leaving it permanently degraded until the next restart.

Edit log segments and why they roll

The edit log is not one growing file. It is a chain of segments, and only the last one is open. A segment named edits_inprogress_<startTxid> is the live tail; when it is closed, it is renamed to edits_<start>-<end> and becomes immutable. That closing operation is called rolling, and almost every interesting behaviour in the checkpoint system is triggered by it.

Segments exist because immutability is what makes the rest tractable. A finalized segment has a known ID range, so a reader can decide whether it needs it without opening it; it can be checksummed once; it can be purged as a unit; and a standby can consume it without worrying that its length will change underneath. The open segment is the only one where length is ambiguous, which is exactly why recovery only ever has to reason about one file.

Rolls happen for several reasons. An operator can force one with hdfs dfsadmin -rollEdits. In HA the standby asks the active to roll on the interval set by dfs.ha.log-roll.period, because a standby that can only read finalized segments needs the active to keep producing them. The active also rolls on its own when the open segment grows disproportionate to the checkpoint threshold, governed by dfs.namenode.edit.log.autoroll.multiplier.threshold and checked on dfs.namenode.edit.log.autoroll.check.interval.ms. And a checkpoint always begins with a roll, since an image can only be cut at a segment boundary.

Enabling dfs.ha.tail-edits.in-progress lets a standby or observer read the open segment rather than waiting for it to close, which collapses tailing lag from the roll period to roughly the poll interval — the prerequisite for observer reads.

The JournalNode as a storage service

A JournalNode is a deliberately small daemon whose entire job is to store edit segments and answer questions about them. It holds no fsimage, no block map, and no namespace in memory, which is why three or five of them can be colocated on hosts that are already doing something else. They are addressed together by a single URI naming the nameservice:

<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://jn1:8485;jn2:8485;jn3:8485/prodns</value>
</property>
<property>
  <name>dfs.journalnode.edits.dir</name>
  <value>/data/jn</value>
</property>

The RPC port defaults to 8485 (dfs.journalnode.rpc-address) and an HTTP port, usually 8480 (dfs.journalnode.http-address), serves segments to readers. Under dfs.journalnode.edits.dir each JournalNode keeps a per-nameservice current/ directory holding the same edits_* segment files as a NameNode, plus a handful of small bookkeeping files — a committed transaction ID, the highest epoch it has promised, the epoch of the writer it last accepted, and a paxos/ directory recording recovery decisions.

A write commits when a majority has fsynced it, which sets the arithmetic of sizing: three JournalNodes tolerate one failure, five tolerate two, and four tolerate only one because three of four is still the majority. The practical consequence is that a JournalNode's fsync latency is on the critical path of every namespace mutation in the cluster, and the committing majority moves at the speed of its slowest member. A JournalNode sharing a spindle with a busy log volume shows up as inexplicable latency on create calls, not as a JournalNode alarm. Timeouts such as dfs.qjournal.write-txns.timeout.ms and dfs.qjournal.start-segment.timeout.ms bound how long the writer waits before treating a member as unavailable.

A JournalNode that was down misses segments. dfs.journalnode.enable.sync lets JournalNodes fetch missing finalized segments from their peers on the interval given by dfs.journalnode.sync.interval, closing gaps in the background instead of leaving a member permanently thin.

Segment recovery — when a writer dies mid-segment

Finalized segments are safe by construction. The interesting case is the open one: a writer died partway through edits_inprogress_99413, and different JournalNodes may hold different amounts of it. One fsynced through transaction 99500, another through 99498, a third was unreachable and has nothing.

Before a new writer may append, it runs a recovery round over that segment. It first claims an epoch higher than any the quorum has seen; each JournalNode that accepts promises never again to honour a request carrying a lower one, and records that promise on disk. With a majority promised, the writer asks each member how much of the in-progress segment it holds, selects the version a majority can agree on, forces the laggards to match it, finalizes the segment at that length, and only then opens a new one. The decision is written into the paxos/ directory so a JournalNode that crashes mid-recovery reaches the same answer on restart.

Two properties fall out of this. First, the cut point is deterministic: whatever a majority durably held is kept, and anything held by only a minority is discarded — which is sound, because a minority write was never acknowledged to a client. Second, the old writer's epoch is now stale, so even if it wakes from a long garbage-collection pause still believing it is in charge, every JournalNode rejects its next append and it can never reach a majority. That is the storage layer's own defence, and it is why journal safety does not depend on anyone being able to reach the failed host. The complementary process-level fencing, the election that decides who gets to attempt recovery in the first place, and the failover time budget are covered in HDFS High Availability.

Who takes the checkpoint, and what fires it

In the classic single-NameNode deployment a separate Secondary NameNode or Checkpoint Node pulled the image and edits over HTTP, merged them, and pushed the result back; that model, and why the merge was offloaded at all, is the subject of NameNode checkpointing. In an HA cluster it is gone. The standby NameNode is already replaying every edit to stay warm for failover, so it is already holding a current namespace in memory and serializing it costs almost nothing extra. Running a Secondary NameNode alongside HA is a configuration error, not a belt-and-braces measure.

The trigger is a race between a transaction count and a clock. dfs.namenode.checkpoint.txns fires a checkpoint once that many transactions have accumulated since the last one, so a busy cluster checkpoints on volume; dfs.namenode.checkpoint.period fires on elapsed time, so a quiet cluster still produces fresh images. Whichever arrives first wins. The checkpointer does not evaluate these continuously — it wakes on dfs.namenode.checkpoint.check.period and tests both conditions, which is why the real interval is always the configured one rounded up to the next check.

The cycle itself is short to describe. The standby asks the active to roll the edit log so there is a clean boundary, replays anything it has not yet applied, writes its in-memory namespace out as fsimage_<txid> with a matching .md5, and uploads it to the active. The active adopts the image, and both sides may now purge history the image has absorbed. On a large namespace the serialization is neither free nor instant, which is one reason the standby's heap must be sized the same as the active's.

Shipping the image — the transfer path and its throttles

The image moves over HTTP between the NameNodes' web ports, through the image-transfer servlet, and it is one of the few places in HDFS where a background maintenance task can hurt a foreground one. A multi-gigabyte fsimage pushed at line rate across a shared network can starve the DataNode heartbeat and client RPC traffic on the same links, and the failure it produces looks like a network problem rather than a checkpoint problem.

dfs.image.transfer.bandwidthPerSec throttles the push, and dfs.image.transfer.timeout bounds how long a stalled transfer may hang before it is abandoned and retried on the next cycle. Bootstrap has its own separate throttle, dfs.image.transfer-bootstrap-standby.bandwidthPerSec, precisely because seeding a new standby is a one-off operation where you usually want the opposite trade-off from the hourly steady state: finish quickly, accept the load. Setting the steady-state throttle and forgetting the bootstrap one is a common way for a routine bootstrapStandby to take an unexpected afternoon.

The .md5 companion file is checked on receipt and again whenever an image is loaded, so a transfer truncated by a timeout is rejected rather than adopted. Compression is available via dfs.image.compress with a codec named by dfs.image.compression.codec; it trades CPU on both ends for wire time and disk, and it makes the image opaque to tools that would otherwise read it directly, so it is a deliberate choice rather than a default worth flipping casually.

Retention and purging — how much history survives

Nothing here is retained forever, and the retention settings are the ones most often left at their defaults until the day they matter. dfs.namenode.num.checkpoints.retained controls how many old fsimage files survive a purge. Keeping more than the minimum is cheap insurance: if the newest image turns out to be corrupt or was produced from a namespace someone had just damaged, an older image plus a longer replay is a recovery path, and having exactly one image is not.

Edit segments are governed by two keys that work together. dfs.namenode.num.extra.edits.retained expresses retention in transactions — keep at least this much history beyond what the newest image already contains — while dfs.namenode.max.extra.edits.segments.retained caps the retention in files, so a cluster that rolls very frequently cannot accumulate an unbounded number of tiny segments. Purging on the JournalNodes follows the same logic; they are told what is safe to discard rather than deciding independently.

The failure mode this creates is specific and worth recognising. A standby that has been down long enough for the segments it still needs to be purged from the JournalNodes cannot resume by tailing, because the history between its position and the current one no longer exists anywhere. It does not fail loudly at startup in a way that suggests the cause; it simply cannot construct a continuous replay. The fix is to re-seed it with bootstrapStandby, not to restart it again and hope. Retention windows are therefore an availability parameter: they set how long a NameNode may be absent before repair becomes re-seeding.

Reading the artifacts offline — oiv and oev

Both artifacts can be opened offline, which is what turns "the namespace looks wrong" from a guess into an investigation. The Offline Image Viewer reads an fsimage without a running NameNode; the Offline Edits Viewer converts a segment into readable XML.

# dump an fsimage to XML (whole namespace, one element per inode)
hdfs oiv -i /data/nn/current/fsimage_0000000000000098765 -o /tmp/ns.xml -p XML

# tab-delimited listing -- path, replication, mtime, blocksize, owner, permissions
hdfs oiv -i fsimage_0000000000000098765 -o /tmp/ns.tsv -p Delimited

# file-size histogram: the fastest way to prove a small-files problem
hdfs oiv -i fsimage_0000000000000098765 -o /tmp/dist.txt -p FileDistribution

# browse an image over a local read-only WebHDFS endpoint
hdfs oiv -i fsimage_0000000000000098765 -p Web -addr 127.0.0.1:5978

# decode an edit segment: every op with its txid, in order
hdfs oev -i edits_0000000000000098766-0000000000000099412 -o /tmp/edits.xml -p XML

# pull the current image off a running NameNode without touching its disks
hdfs dfsadmin -fetchImage /tmp/imagebackup

The FileDistribution processor deserves particular mention because it answers a question people usually try to answer with a recursive hdfs dfs -ls that hammers the NameNode for hours. Reading it from an image costs the NameNode nothing at all. The Web processor is similarly useful for after-the-fact forensics: you can walk a namespace as it existed at a past checkpoint.

Two caveats. Images written by modern Hadoop use a protobuf-based format, and very old images need hdfs oiv_legacy instead. And oiv on a large image is memory-hungry for some processors — run it on a workstation with a copy of the file, not on the NameNode host where it will compete with the process you are trying to keep healthy.

The operations that touch these files

A handful of commands are the ones that actually touch these files, and each has a narrow correct use.

# seed a new/rebuilt standby from the active's latest image, then tail the journal
hdfs namenode -bootstrapStandby

# one-time: copy existing local edits into a freshly configured JournalNode quorum
hdfs namenode -initializeSharedEdits

# force a fresh checkpoint on a running NameNode (safe mode required)
hdfs dfsadmin -safemode enter
hdfs dfsadmin -saveNamespace
hdfs dfsadmin -safemode leave

# close the open segment now -- lets a lagging standby make progress
hdfs dfsadmin -rollEdits

# last resort: interactively skip past a corrupt edit record on startup
hdfs namenode -recover

bootstrapStandby is the routine one. It contacts the active, downloads the newest fsimage into the local storage directories, and leaves the node ready to tail the journal — the correct response both to commissioning a second NameNode and to a standby that fell outside the retention window. It refuses to run over a populated storage directory unless forced, which is a guard worth respecting rather than flagging past.

saveNamespace is how you take a checkpoint on demand, typically before an upgrade or before deliberately restarting a NameNode whose checkpointer has been broken; requiring safe mode is what makes the resulting image consistent. namenode -recover is genuinely a last resort — it exists for a truncated or corrupt edit tail, it discards records to make startup possible, and discarding acknowledged namespace edits is data loss. Copy the storage directory before running it. In a healthy HA cluster you should never need it, because the quorum is what makes a torn local tail irrelevant.

Metrics, failure modes, and the standing discipline

The artifacts described here expose themselves through JMX, and the useful alerts are all derived rather than obvious. Checkpoint freshness is the first: LastCheckpointTime and TransactionsSinceLastCheckpoint on the NameNode tell you whether the checkpointer is actually running. This matters because a broken checkpointer is invisible during normal operation — the active serves from RAM and does not care how long the log is — and reveals itself only as a restart that takes forty minutes instead of forty seconds. Alert on checkpoint age exceeding a small multiple of dfs.namenode.checkpoint.period, not on a restart that has already gone wrong.

Standby lag is the second: compare LastAppliedOrWrittenTxId across both NameNodes and watch the gap rather than the absolute value. A growing gap means slow promotion later and usually points at edit tailing or a JournalNode, not at the standby itself. TransactionsSinceLastLogRoll tells you whether rolling is keeping up, and FsImageLoadTime records what the last startup actually cost — the single best predictor of the next one.

On the JournalNodes, treat fsync latency as a first-class metric. The sync-latency percentiles and operation counts each JournalNode publishes are the early warning for the storage problem that will otherwise surface as cluster-wide write latency with no obvious cause. Watch the promised and last-writer epoch values too: an epoch climbing when no failover was performed means something is repeatedly attempting recovery, which is a symptom worth chasing before it becomes an incident.

The failures that hurt in this subsystem are rarely dramatic at the moment they occur. A checkpointer that stopped a week ago produces no symptom at all until the day someone reboots a NameNode and safe mode runs for the length of a meeting. A JournalNode on a contended disk produces no JournalNode alert, only a cluster where create got slower. A standby that was left down over a long weekend produces no warning until it cannot rejoin. In every case the artifact-level metric was available and unwatched.

The corresponding discipline is short. Keep more than one fsimage. Give the JournalNodes their own storage rather than sharing a busy volume. Set retention windows against how long a NameNode might realistically be out for repair, not against disk convenience. Take a deliberate checkpoint before upgrades. Copy the storage directory before any command with "recover" in its name. And back up an fsimage off-cluster with -fetchImage, because a quorum protects you from hardware failure and from nothing else — a mistaken recursive delete is replicated to a majority of JournalNodes within a second, which is what snapshots exist for.

Read further along the same seam: the NameNode itself for what the namespace in memory looks like, safe mode for what a restart is waiting on once replay finishes, and the small files problem for why the size of that namespace is the constraint behind most of these numbers.