HDFS Safe Mode — Read-Only Startup State and When to Leave It

What safe mode actually is

Safe mode is a flag on the NameNode that changes which RPCs it will honour. Metadata reads are served normally: ls, stat, du, opening a file and streaming its bytes from DataNodes all work, assuming the replicas involved have checked in. Everything that mutates the namespace is rejected with an exception naming safe mode: create, append, truncate, rename, delete, mkdir, setrep, quota changes, permission changes, snapshot creation. Clients see a hard failure rather than a queue, so a job that writes will die rather than hang.

The second half of the flag is the half operators forget. While it is set, the NameNode also suspends its own background work on the block map. It does not put anything on the replication queue, it does not issue invalidations to delete surplus replicas, and it does not act on the difference between the replication factor a file was created with and the number of replicas it can currently see. Those two halves exist for the same reason. A write would be recorded against a namespace whose physical backing is unverified; a background repair would be scheduled against a census that is still being taken.

A useful way to hold it: the NameNode in safe mode is a filesystem that can answer questions about itself but has not yet earned the right to change anything, including the things it would normally change on its own initiative.

Why block locations are the thing being waited for

HDFS metadata splits into two halves with completely different durability stories, and safe mode exists entirely because of the split.

The first half is the namespace: the directory tree, file names, ownership and permissions, replication factors, and the ordered list of block IDs that make up each file. All of that is written down. It survives in the fsimage plus the edit log that has accumulated since the last checkpoint, and a restarting NameNode can reconstruct it exactly, with no help from anyone. (What determines how long that reconstruction takes, and why a stalled checkpointer turns a restart into an outage, is developed in HDFS NameNode checkpointing.)

The second half is the block location map: for block blk_1073741900, which DataNodes currently hold a copy. This is never written to disk on the NameNode, and the omission is deliberate rather than an oversight. A location is not a fact about the NameNode, it is a claim about the contents of some other machine's disk right now. Persisting it would only preserve a claim that may have expired while the NameNode was down: disks fail during maintenance windows, machines get reimaged, a rack loses power and comes back empty. A stale location file would be worse than no file at all, because the NameNode would believe it.

So the state a NameNode wakes up in is asymmetric. It knows every file and every block ID those files are made of. It knows nothing about where any of those blocks physically are. Each DataNode closes that gap by sending a full enumeration of the blocks on its volumes shortly after it registers, and only when a sufficient share of the expected block IDs have been claimed by some live DataNode is it reasonable for the NameNode to start making decisions.

The exit condition, precisely

Loading the fsimage and replaying the edits are phases, not conditions. They must finish before the NameNode can even name the blocks it is waiting for, but finishing them does not release anything. The condition proper is arithmetic over two counters that only start moving once the namespace is in memory.

The denominator is the total number of blocks the namespace says should exist. The numerator is the number of those blocks that have been claimed by at least the minimum replica count, governed by dfs.namenode.replication.min, which is 1 by default. One live copy is enough for a block to count as satisfied here; safe mode asks whether data exists at all, not whether it is adequately protected. A separate override, dfs.namenode.safemode.replication.min, exists if you want the safe-mode arithmetic to use a different bar from the one the replication monitor uses.

The ratio must reach dfs.namenode.safemode.threshold-pct, which defaults to 0.999. The value is not 1.0 on purpose: on a cluster of any size there is nearly always a handful of blocks whose only replicas are on a machine that is down, and demanding perfection would mean a cluster that never becomes writable after a single dead node. The two boundary values are worth knowing because both get used deliberately. Set the threshold to zero or below and the NameNode never enters safe mode at startup at all, which is a reasonable choice for a scratch cluster and a bad one anywhere else. Set it above 1.0 and the condition can never be satisfied, which pins the NameNode in safe mode until an operator explicitly releases it.

A second gate, dfs.namenode.safemode.min.datanodes, requires a floor number of live DataNodes independent of block coverage. It defaults to 0, meaning no floor. It is worth setting on a cluster where a small subset of nodes stores enough of the block population to satisfy the percentage on its own, since without it a cluster can go writable with most of its capacity absent.

Why there is an extension period

When the ratio crosses the threshold, the NameNode does not exit immediately. It starts a timer, dfs.namenode.safemode.extension, and exits only when the timer expires with the condition still holding. The default is 30 seconds.

The reason is that the instant of crossing is the worst possible moment to start acting. Reports arrive in a burst, so the counter typically leaps over the line rather than easing across it, and the reports still in flight behind it carry a large share of the remaining locations. Exiting on the exact tick would mean initialising the replication queues from a snapshot taken at the earliest legal moment, then discovering over the next several seconds that a substantial number of blocks flagged as under-replicated actually had copies all along. The extension window costs half a minute and buys a block map that has stopped moving. It also acts as a damper against the ratio oscillating around the threshold as blocks are both reported and invalidated.

The startup sequence

Read the NameNode log during a boot and the phases are distinct, which matters because each one fails differently and only one of them is the one people mean when they say the cluster is stuck.

The NameNode first reads the most recent fsimage from a local name directory and materialises the inode tree in the heap. It then fetches every edit transaction newer than that image, from the local edits directory or from the JournalNodes, and applies them in order until it reaches the last committed transaction ID. Only now does it have a complete namespace, and therefore a complete list of the block IDs it is missing locations for. It opens its RPC ports, DataNodes register, and each sends a full block report enumerating what it holds. The NameNode reconciles each report against the expected block set, and the satisfied-block counter climbs. When it crosses the threshold the extension timer runs, and when the timer expires safe mode clears.

What happens in the seconds after the flag drops is the payoff for the wait. The NameNode scans the block map, builds the under-replication queues, identifies surplus replicas for invalidation, and quarantines blocks whose reported generation stamps or lengths contradict the namespace. All of the corrective work the cluster needs is scheduled at once, from a census that is now essentially complete. Deletion is usually held back a little longer than replication is, via a startup deletion-delay knob, so that a DataNode that registers late does not have its blocks discarded on the strength of a report the NameNode had not received yet.

One lever worth naming here is dfs.blockreport.initialDelay. It defaults to 0, so out of the box every DataNode sends its first full report as soon as it can, and on a large cluster that means thousands of reports arriving in the same few seconds to be processed under the namespace write lock. Raising it spreads the arrivals over a randomised window, which smooths the lock contention at the cost of pushing out the moment the threshold is reached. It is a trade between a shorter but spikier startup and a longer but steadier one.

Manual safe mode behaves differently from automatic

The operator interface is four subcommands, and one of them lies about being symmetrical with the automatic behaviour.

# is it on, and if so why
hdfs dfsadmin -safemode get

# put the NameNode into safe mode and keep it there
hdfs dfsadmin -safemode enter

# release it
hdfs dfsadmin -safemode leave

# block this shell until safe mode is off (returns immediately if it already is)
hdfs dfsadmin -safemode wait

# release it while ignoring the threshold entirely
hdfs dfsadmin -safemode forceExit

Automatic safe mode is a condition. The NameNode holds it because the arithmetic is unsatisfied and releases it the moment the arithmetic is satisfied and the extension has elapsed. Manual safe mode is a latch. Once you have issued enter, the NameNode stays in safe mode no matter how healthy the block map becomes; the threshold being met is no longer an exit path, and only an explicit leave will clear it. This is the correct design for a maintenance freeze, and it is a reliable way to leave a cluster read-only for hours if you enter safe mode from a script and the script exits before its cleanup step.

The inverse trips people just as often. Issuing leave during a startup that had not finished does not merely skip the current wait; it turns off automatic safe mode for that NameNode process. The block-report counter keeps climbing but nothing is watching it any more, and you do not get the protection back for that boot unless you deliberately re-enter.

-safemode wait is the one to reach for in automation. It gives you an ordering primitive rather than a poll loop, so a startup script that must not launch jobs against a half-warm cluster can simply block on it. Legitimate uses of enter are narrower than they look: freezing the namespace so that a consistent fsimage can be captured (saving the namespace requires safe mode), pinning a cluster read-only ahead of a risky upgrade step, and stopping a runaway ingestion pipeline at the filesystem when stopping it at the source is slower.

What breaks when you force a leave

Forcing an exit does not manufacture the missing information. It tells a NameNode that knows it is ignorant to start acting anyway, and the consequences follow in a specific order.

First, every block nobody has claimed is now simply missing as far as the NameNode is concerned. Reads of files containing those blocks fail rather than blocking, and any job that touches them fails with them. Second, the replication queues initialise from the partial map, so blocks whose replicas sit on DataNodes that have not reported yet look under-replicated, and the NameNode starts copying them across the cluster to fix a problem that does not exist. On a cluster where a rack is still booting, that is a large fraction of the namespace being re-replicated at once, saturating exactly the network the absent DataNodes need in order to register. Third, when those DataNodes do finally report, their copies arrive on top of the ones just created, the affected blocks flip from under-replicated to over-replicated, and the NameNode issues invalidations to trim them. The cluster has now moved the same data twice and ended where it started.

Underneath all of that, the namespace is writable while the picture is wrong, so a user deleting a directory whose blocks are temporarily unaccounted for gets exactly what they asked for, permanently.

None of which makes forcing an exit wrong in every case. It is the right call when you have already established that the missing data is genuinely gone, from an evacuated rack or reimaged hosts, and the cost of leaving the cluster read-only exceeds the cost of the churn. The distinction is whether you are forcing an exit because you have finished diagnosing or because you have not started.

Diagnosing a cluster that will not leave

Before anything else, ask the NameNode. Both the web UI and -safemode get print which specific condition is unmet, including the reported block count, the total expected, the threshold in force and the live DataNode count against the required minimum. That message tells you which of the following you are in, and the cases have almost nothing in common except the symptom.

# how many DataNodes are actually live, and how much capacity they carry
hdfs dfsadmin -report

# blocks with no valid replica anywhere, and the files that own them
hdfs fsck / -list-corruptfileblocks

# full health report for a subtree, with block IDs and current locations
hdfs fsck /warehouse/prod -files -blocks -locations

# ask a DataNode to re-send its full report immediately
hdfs dfsadmin -triggerBlockReport <datanode_host>:<ipc_port>

Reported blocks far below total

A large shortfall means DataNodes are not talking, not that data is lost. Compare the live count from -report against what you expect to be running. The usual causes are mundane: the DataNode processes never started, hostname or DNS changed under them so their registration is rejected, Kerberos tickets or keytabs expired so authentication fails silently from the operator's point of view, or a firewall rule is blocking the NameNode's RPC port. One cause is easy to misread: a DataNode whose failed volume count exceeds its tolerance shuts itself down rather than running degraded, so a batch of bad disks presents as missing nodes rather than as disk errors.

Reported blocks just short of the threshold

When the ratio has stalled at, say, 0.9962 against a threshold of 0.999 with every DataNode live, the shortfall is real data loss and no amount of waiting fixes it. Use -list-corruptfileblocks to enumerate the affected files, then decide per file. Data that has an upstream source or a backup should be restored and the damaged copies removed. Data that is genuinely gone has to be reaped with fsck -delete, or moved aside with fsck -move, before the ratio can climb; either way the decision is a data decision, not a filesystem one, and it belongs to whoever owns the dataset.

Block coverage fine, DataNode floor not met

If the block ratio has cleared the threshold but the message complains about the number of live nodes, you have hit dfs.namenode.safemode.min.datanodes. This is the gate doing its job on a cluster whose block population is concentrated enough to satisfy a percentage without satisfying an operator, and the fix is to bring nodes back, not to lower the floor.

Nothing to do with blocks at all

A NameNode also enters safe mode at runtime when free space in its name directories drops below its configured reserve, because it will not risk a partial edit write. This one confuses people badly, since it strikes a healthy running cluster and the block counters look perfect. The tell is the log line about inadequate resources rather than about block reports. Free the disk, usually by trimming retained edit segments or old fsimages, and it clears on its own.

Not stuck, just slow

The most common diagnosis on a large cluster is that nothing is wrong. Watch the log for fsimage load progress and edits replay progress and confirm the transaction counter is advancing, or watch the reported block count climb between two -safemode get calls. If it is moving, the answer is to wait, and forcing an exit here converts a slow start into a genuine incident.

Safe mode in an HA pair

Safe mode is per NameNode, not per cluster. Each NameNode in a highly available pair keeps its own flag and its own satisfied-block arithmetic, and neither one can clear the other's.

This works out well because DataNodes register with and report to every configured NameNode, so a standby that has been up for a while already holds a warm block map and has long since left safe mode on its own. (The mechanics of the standby staying current, and of failover itself, belong to HDFS high availability and NameNode HA architecture.) The consequence for safe mode is that a failover between two long-running NameNodes involves no safe-mode window at all, which is the single biggest reason a rolling NameNode restart beats a simultaneous one: restart them one at a time and the cluster never has a moment when the node serving clients is waiting on block reports.

The case to be careful about is a standby that transitions to active while it is still in safe mode, which happens when both NameNodes were restarted close together. The transition does not waive the wait. The new active serves reads and rejects writes until its own condition is satisfied, exactly as a lone NameNode would, and that is the correct behaviour even though it looks like a failover that did not take.

One operational wrinkle: hdfs dfsadmin -safemode in an HA cluster addresses every NameNode in the nameservice, so a get returns a line per node and an enter latches all of them. Targeting a single NameNode means passing its address explicitly with -fs, which matters when you intend to freeze one node and not the pair.

How long the wait scales

Startup duration is dominated by namespace size, and each phase scales against a different quantity, which is why the fix depends on which phase is slow.

Reading the fsimage scales with the number of namespace objects, files plus directories plus blocks, since every one of them becomes a heap structure. Replaying edits scales with the number of transactions written since the last checkpoint, so its duration is a property of checkpointing health rather than of cluster size; a cluster whose checkpointer has been quietly failing will replay days of transactions and turn a routine restart into a long one. Waiting for block reports scales with the total replica count across the cluster and with how much of that arrives at once, because each report is reconciled while holding the namespace write lock. A cluster of a few million objects clears all three phases in well under a minute. Hundreds of millions of objects put you into tens of minutes, and the curve is unforgiving because all three terms grow together.

The levers, in the order they usually pay off: keep the checkpointer healthy and alarm on checkpoint age, so the replay term stays small. Keep the object count down, since the small files problem inflates the fsimage load and the report volume simultaneously, one object at a time. Put the name directories on fast local storage, because the fsimage read is a bulk sequential load that a slow disk will dominate. Spread the report arrivals with the initial delay if lock contention rather than raw volume is the bottleneck. Note also that erasure coding changes the report arithmetic, because it changes how many stored objects a given volume of data becomes.

Measure it deliberately rather than discovering it during an incident. Restart time from process start to safe mode off is a number every cluster should have, tracked over time, because it is the length of the write outage you have already agreed to accept.