Why it matters

Startup time is a huge operational concern for NameNodes. A cluster whose NameNode takes an hour to restart is one where every planned maintenance window is a service outage. Fast checkpoints and small edit logs keep restart time in the minutes range even for clusters with billions of edits in history.

Durability guarantees are the other concern. A cluster that loses metadata after a crash has effectively lost all its data, because the block map that maps files to physical block locations was in memory. Reliable checkpointing and quorum-based journal storage are what turn HDFS from a research project into an enterprise database.

Advertisement

The architecture

The FsImage is a binary serialization of the entire NameNode namespace. It contains every inode, every file-to-block mapping, and every permission. It is written to local disk periodically, typically hourly. Once written, it is immutable until the next checkpoint replaces it.

The edit log is a sequence of mutation records: create file, delete file, rename, append, permission change. Every RPC that modifies namespace state appends a record. In HA setups, the edit log lives on a JournalNode quorum of three or five nodes. Every edit record must be acknowledged by a majority of JournalNodes before the NameNode considers the mutation durable.

Active NameNodewrites editsStandby NameNodetails edits + checkpointsJournalNodes (quorum of 3-5)distributed edit logedit streamread streamStandby periodically produces fresh FsImage and pushes to ActiveEvery checkpoint period (~1h): serialize namespacepush FsImageActive persists it
Namespace durability: JournalNodes hold edit log, standby periodically produces fresh FsImage.
Advertisement

How it works end to end

On startup, the NameNode loads the most recent FsImage from local disk, then replays every edit log record newer than the FsImage from the JournalNode quorum. When replay completes, the in-memory namespace matches what was there before the shutdown. The NameNode is then ready to serve client RPCs.

The checkpoint role is played by the standby NameNode in HA setups. It continuously tails the edit log and applies each mutation to its own in-memory namespace. Periodically it serializes its namespace to a new FsImage and pushes it to the active NameNode's local disk. This means the active NameNode always has a fresh FsImage without ever taking a checkpoint itself. Truncating the edit log to keep it small also happens as part of this cycle.

In non-HA setups (increasingly rare in production), the checkpointing role is played by a Secondary NameNode or Checkpoint Node, which is essentially a subset of what the HA standby does. The pattern is the same but the naming is legacy.