Why architecture matters here
The architecture matters because the naive alternatives to point-in-time protection are unusable at HDFS scale. Copying a directory to make a backup doubles storage and takes as long as reading all the data — hours to days for large trees — during which the source keeps changing, so the copy is never consistent. Stopping writers to get a clean copy is a non-starter for a cluster that ingests continuously. Snapshots sidestep both: creation is instant and consistent because it captures metadata under a lock, and it consumes no space at creation because it shares blocks. That combination is what makes frequent, routine protection affordable — you can take hourly snapshots of a critical directory without thinking about the cost.
The immutability of HDFS blocks is the enabling invariant, and it is worth internalizing. Because a block, once finalized, is never modified in place, the exact bytes a file had at time T are still on disk after T as long as something references them. A snapshot is that reference. There is no need to copy data to preserve the past, because the past's data was never overwritten — it was only dereferenced by the live tree, and the snapshot keeps a reference alive. Copy-on-write falls out of this naturally: a new write produces a new block, the live file points at it, and the snapshot keeps pointing at the old one.
Consistency is the second reason the architecture is valuable. A snapshot captures the namespace atomically from the NameNode's perspective: every file and its block list as of a single instant. That gives operators a coherent view to restore from — no half-written state, no torn set of files where some reflect the new version and some the old. For workloads that produce a directory of related outputs, being able to roll the whole directory back to one consistent moment is far more useful than per-file versioning that can leave the set internally inconsistent.
Finally, snapshots enable efficient incremental data movement, which is the backbone of real backup and replication. The snapshotDiff operation reports exactly which files were created, deleted, modified, or renamed between two snapshots. Feeding that diff to distcp -update -diff lets a backup job copy only what changed since the last run rather than rescanning the whole tree — turning cross-cluster replication of a petabyte-scale dataset from a full sweep into a small delta. The snapshot mechanism is thus not only local rollback protection but the primitive on which incremental disaster-recovery pipelines are built.
The architecture: every piece explained
The snapshottable directory is the unit of the feature. An administrator runs hdfs dfsadmin -allowSnapshot /data to mark a directory as snapshottable; only then can snapshots be taken of it. A directory can hold up to 65,536 simultaneous snapshots, and nested snapshottable directories are disallowed to keep the ownership of shared blocks unambiguous. The flag is deliberately explicit so operators control which subtrees carry the metadata overhead.
The snapshot itself is exposed through a virtual .snapshot directory. After hdfs dfs -createSnapshot /data s1, the captured state is readable at /data/.snapshot/s1/... using ordinary file-system operations. The path is read-only and does not appear in normal directory listings, so it does not clutter the namespace, but it lets any tool — cat, a MapReduce job, distcp — read the past exactly as it read the present.
The NameNode diff lists are where the cleverness lives. The NameNode does not store a full copy of the tree per snapshot; it stores the live tree plus, for each snapshot, a diff describing how the live tree has changed since that snapshot was taken — files created, deleted, modified, and directory entries renamed. Reading a file through .snapshot/s1 means applying those diffs in reverse to reconstruct what the tree looked like at s1. This keeps snapshot metadata proportional to change rather than to tree size, mirroring the block-sharing story at the namespace level.
Block sharing and refcounted deletion are the data-plane counterpart. When a snapshot is created, the blocks of every captured file are simply referenced by both the live inode and the snapshot's view — one physical copy, two logical owners. When a file is later overwritten, new blocks are written for the live version while the snapshot retains its pointers to the old blocks. When a file is deleted from the live tree but a snapshot still needs it, its blocks move into the snapshot's ownership rather than being freed. The block manager only instructs DataNodes to physically delete a block when no snapshot and no live file references it any longer — classic reference counting. The snapshotDiff API exposes the diff lists to clients, and quotas interact with all of this: space held by snapshots still counts against a directory's space quota, which is the mechanism by which retained snapshots can, surprisingly, cause new writes to fail. Together — the snapshottable flag, the .snapshot path, the diff lists, block sharing, refcounted deletion, and the diff API — form the complete architecture.
End-to-end flow
Walk the lifecycle. An operator enables snapshots on /data/warehouse and, as part of a nightly job, runs createSnapshot warehouse backup-0712. The NameNode takes a lock on the subtree just long enough to record the snapshot: it creates a snapshot entry and initializes an empty diff — nothing has changed yet, so the snapshot and the live tree are identical and share every block. The whole operation is milliseconds and consumes essentially no disk. Writers that were appending to files in the tree resume immediately.
Over the following day the warehouse changes. A job overwrites a partition file: HDFS writes new blocks for the new content, the live inode now points at the new blocks, and the NameNode records in backup-0712's diff that this file was modified, keeping the snapshot's pointer to the old blocks. Another job deletes an obsolete file; because the snapshot captured it, its blocks are not freed — ownership passes to the snapshot and the deletion is recorded in the diff. A brand-new file created after the snapshot is invisible to backup-0712 and lives only in the live tree. Disk used by the snapshot has now grown by exactly the size of the overwritten and deleted data — the cost of divergence.
Now a mistake happens: an engineer accidentally deletes an entire important partition directory. Because a snapshot exists, the data is not gone — it is readable at /data/warehouse/.snapshot/backup-0712/partition=2026-07-11/. Recovery is a copy from the snapshot path back into the live tree: hdfs dfs -cp or a distcp from the .snapshot source restores the files, pointing the live tree back at blocks that were never physically deleted because the snapshot held their references alive. There was no restore-from-tape, no recompute — just re-linking to preserved blocks.
Finally, the backup pipeline uses two snapshots to move only deltas. The next night's job creates backup-0713, then runs hdfs snapshotDiff warehouse backup-0712 backup-0713 to get the exact set of creates, deletes, modifies, and renames since yesterday. It feeds that to distcp -update -diff backup-0712 backup-0713 against the DR cluster, which applies only those changes to its copy — a few gigabytes of delta instead of a full re-scan of the petabyte warehouse. Once the DR side confirms, the pipeline can delete backup-0712; its uniquely-held blocks are refcounted down and reclaimed by the block manager, and the cycle repeats. Instant creation, cost-on-divergence, in-place recovery, and diff-driven replication are the four behaviors that make snapshots the workhorse of HDFS data protection.