HBase Snapshots — Instant Point-in-Time Table State for Backup and Recovery

Why it matters

Backup is a compliance requirement for most enterprise HBase deployments. Snapshots make backups fast (instant capture) and cheap (no data copy). This makes daily or hourly snapshots practical, whereas the naive approach of scanning and exporting a table can take hours per backup.

Snapshots also enable safe experimentation. Clone a snapshot into a new table, run risky operations against the clone, and delete it when done. The original is untouched.

A snapshot is a manifest of file names, not a copy of data

The first thing that confuses people is that snapshot 'orders', 'orders-20260805' returns in a couple of seconds against a 40 TB table. Nothing was copied, because nothing needed to be. HBase writes store files once and never edits them in place: a MemStore flush produces a new HFile, a compaction produces a new HFile and retires its inputs, and no process ever rewrites bytes inside a file a reader might have open. That immutability is the whole trick. To freeze a table you only have to write down which files were current at the moment you looked.

The output lands under the HBase root at .hbase-snapshot/<name>/: a small descriptor recording the snapshot name, the source table, the creation time and the snapshot type, plus a manifest that enumerates, per region, the region boundaries, the column families, and the store file names in each family. A thousand-region table produces a few hundred kilobytes of protobuf. Creation cost is proportional to the number of files, not to the number of bytes, and every one of those files is already on disk and already replicated.

Everything else about snapshots is a consequence of that one design decision. They are near-instant because the operation is a listing. They stay valid indefinitely because the listed files are immutable. They cost nothing at creation and a great deal later, because a listing is a standing claim on files the storage layer would otherwise be free to delete. For what is actually inside those HFiles - trailer, multi-level block index, bloom blocks, data block encoding - see the HFile format; this article is about the layer that points at them.

The architecture

A snapshot is a metadata operation. When you snapshot a table, HBase captures the current set of HFiles for each region, plus the current memstore contents (flushed to a temporary HFile). The snapshot is stored as a directory of manifest files pointing to the existing HFiles.

Because HFiles are never modified, the snapshot remains valid indefinitely. However, if the table would normally delete an HFile (during compaction cleanup), and the snapshot still references it, the HFile is kept until the snapshot is deleted. This means snapshots consume storage proportional to how much data has been rewritten by compactions since the snapshot was taken.

Flush versus skip-flush, and the consistency you actually get

Start from the fact that governs everything here: a snapshot does not contain the WAL. Nothing under .hbase-snapshot can replay an edit. The only mutations a snapshot holds are mutations that had already been written into an HFile when the manifest was taken. That single property explains the two modes, the loss window, and why restore is not a rewind.

The default mode flushes. Each RegionServer is told to push its MemStore to disk before recording its file list, so every edit the cluster had accepted up to that moment becomes a file the manifest can name. The alternative is snapshot 'orders', 'orders-fast', {SKIP_FLUSH => true}, which skips that step. Whatever is sitting in the MemStore is then simply absent from the snapshot - not deferred, not recoverable from it later by any means. Those edits are still safe in the live table, because they are in the WAL and the table's durability level governs that independently. The snapshot just does not have them.

What the guarantee actually says

Even in flush mode, the guarantee is per-region, not a cluster-wide transactional instant. Each region is captured consistently as of its own moment inside the coordination barrier, and those moments can be a few hundred milliseconds apart across a large table. HBase never offered cross-row atomicity in the first place, so this is rarely the thing that bites you - but a snapshot taken during a burst of related writes spread over several regions can capture a state no single client ever observed. If that matters for your data model, the answer is to quiesce writes for the duration, not to pick a different snapshot flag.

When skipping the flush is the right call

The flush is not free. Forcing every region of a large table to flush produces a burst of small files that then have to be compacted, so an hourly snapshot schedule in flush mode is also an hourly bump in compaction load. Skip-flush is the right choice when the table is read-mostly or bulk-loaded and the MemStore is close to empty anyway, or when the snapshot exists to feed an analytics job that does not care about the last few minutes of writes. Use flush mode when the snapshot is a recovery point, because a recovery point that silently omits recent writes is worse than no recovery point.

How the master coordinates the capture, and what a failure does

Taking a snapshot of an online table requires agreement from every RegionServer hosting a region of it, and the master drives that as a two-phase barrier. In the prepare phase each participating server is told which of its regions to capture; it flushes if the mode requires it, writes a per-region manifest naming the store files in each family, and acknowledges. Only after the master has collected an acknowledgement from every member does it commit.

The commit itself is refreshingly plain, and it is what makes the operation all-or-nothing. The snapshot is assembled in a working directory, and the final act is moving that directory into the completed snapshot directory. In HDFS a rename is a single NameNode metadata operation. Before it happens, the snapshot does not exist as far as list_snapshots and every consumer is concerned; after it happens, it exists complete. There is no window in which a half-built snapshot is visible and restorable.

So the failure mode is a failed command, not a corrupt snapshot. If any member errors, times out, or dies mid-barrier, the master aborts, the working directory is cleaned up, and nothing is published. On a busy cluster the most common cause of a spurious failure is region movement during the barrier - a region reassigned by the balancer while its old host was still preparing. The correct response is to retry, once the region is settled and, if a server actually died, once its WAL has been split and its regions reopened. Retrying blind in a loop during an assignment storm just extends the storm.

One honest caveat on mechanism: the online-snapshot barrier is its own coordination path, and which subsystem carries it has moved between releases. The master's general durable state-machine framework for assignment and DDL is described in Procedure v2; do not assume the two are the same thing, and reason about snapshots at the level of the observable semantics above, which have not changed.

HFileLink, the archive directory, and the cleaner interlock

The problem the archive solves

A snapshot manifest names files by their live path. Compaction's entire purpose is to make those names go away - merge ten files into one, retire the ten. If retirement meant deletion, every snapshot would rot the moment the next major compaction ran, and snapshots would be useless.

So HBase never deletes a store file directly. When a compaction retires its inputs, they are moved to /hbase/archive/data/<ns>/<table>/<region>/<family>/, a path that mirrors the live layout exactly. The live table stops seeing them at once; the bytes remain. A master-side cleaner chain then decides, asynchronously and much later, whether an archived file may actually be removed - and a file is removed only if every cleaner in the chain agrees.

HFileLink: a reference that survives the move

Because an archived file lives at a different path from a live one, a snapshot cannot hold a plain path. HFileLink is the indirection. A link records the original table, region and file it refers to, and resolves at open time by checking the live location first and the archive location second. The reader that opens it neither knows nor cares which one answered. The same mechanism is what a cloned table's store directories contain: links, not data.

Links also need a reverse index, because the cleaner has to answer "does anything still point at this archived file?" cheaply. Scanning every snapshot manifest and every cloned table's regions on every cleaner pass would not scale, so HBase maintains back-reference entries next to the linked file recording who refers to it. This bookkeeping is normally invisible, and the rare ugly failure - a clone whose backing files were reaped underneath it - is what happens when it is lost, usually through someone moving or deleting files under /hbase by hand.

The interlock, and why it is the same shape everywhere

SnapshotHFileCleaner is the plugin that vetoes deletion of any archived file a snapshot manifest still names. It sits in the same master cleaner chain as the time-to-live cleaner that ages out genuinely unreferenced archive files, and - on a cluster running replication - the log cleaner that protects WALs a peer has not yet shipped. Every one of them is the same shape of interlock: a component that still needs a file blocks its removal, and a component that stops answering blocks it forever. A snapshot nobody remembers taking is precisely that - a permanent veto over a set of archived files, held by nobody who knows they are holding it.

How it works end to end

Snapshot creation flow: the HMaster coordinates all RegionServers hosting regions of the target table. Each RegionServer flushes its memstore (making all in-memory writes durable), then records the set of HFiles that make up its regions. The manifest is written to HDFS. Total time is dominated by the flush, typically seconds.

Restore replaces the current table with the snapshot version. All post-snapshot changes are discarded. Restore is destructive — practice it in staging first.

Clone creates a new table from the snapshot without touching the original. The clone shares HFiles with the original via HDFS hardlinks (or references), so no data is copied. Compactions on the clone eventually rewrite the HFiles, at which point the clone starts using independent storage.

Export ships a snapshot to another HDFS cluster, copying HFile data over the network. This is how you transfer backups off-cluster or seed replication targets.

The storage cost that arrives later

Do the arithmetic on a 10 TB table. Snapshot it at 09:00 and its marginal cost at 09:01 is the manifest - kilobytes. Then a major compaction sweeps the table. Every input file is rewritten into a new output file and the inputs are archived, but the snapshot still names them, so SnapshotHFileCleaner refuses to let them go. You are now storing 20 TB: 10 TB live and 10 TB archived, and HDFS is replicating all of it three times.

That is the worst case, and it is not exotic. A table on a weekly major compaction cycle reaches it every week for any snapshot older than the last compaction. The general rule is simple enough to put on a runbook: a snapshot's real storage cost is the volume of data rewritten since it was taken. Snapshots of a cold, append-only table cost almost nothing to keep forever. Snapshots of a table under heavy overwrite and compaction churn approach a full second copy per snapshot generation.

Overlapping snapshots union rather than multiply. Ten daily snapshots of a slowly-changing table may pin barely more than one snapshot's worth of files, because they name mostly the same files. Cost tracks the set of distinct pinned files, not the number of snapshots - which is also the reason deleting nine of ten snapshots can free precisely nothing, if the tenth still names everything the others did.

Why your capacity dashboard misses it

The reason this catches teams by surprise is a monitoring artifact rather than a subtlety of the mechanism. hdfs dfs -du -s /hbase/data/<ns>/<table> reports the live table and nothing else. The pinned bytes are under /hbase/archive, a completely separate path. Any capacity dashboard keyed on the table directory - and most are, because that is the number that maps to a tenant - underreports by exactly the amount the snapshots are holding. The numbers worth graphing are du on the archive path and total HDFS usage against the sum of live table sizes. A gap that only ever grows is snapshot retention, not a leak, and no amount of compaction tuning will close it.

clone_snapshot as copy-on-write, restore_snapshot as destruction

Two operations consume a snapshot, and they are not variations on a theme - one is safe and one is not.

clone_snapshot materialises a new table from the snapshot. It gets a new name and its own region directories, but the store files inside those directories are HFileLinks pointing back at the original files. Creation is therefore a metadata operation: near-instant, no bytes copied, regardless of table size. The clone is a real table - writable, splittable, compactable, independently configurable. It behaves as copy-on-write in the practical sense: new writes land in the clone's own files, and as the clone compacts, links are progressively replaced by real data until it owns independent storage. Drop the source table afterwards and the clone keeps working, because the archive holds the linked files on its behalf.

restore_snapshot takes the table back to the snapshot's contents in place. Everything written since the snapshot is gone - not moved somewhere retrievable, gone from the table. Since a snapshot carries no WAL, there is no mechanism that could replay the missing interval; restore is a replacement, not an undoable rewind. The table has to be disabled first, so it is also an outage on that table for the duration.

The practical protocol follows directly. Prefer clone. To check whether a snapshot contains what you think it does, clone it and query the clone. To recover fifty deleted rows, clone the snapshot and copy those rows across - do not restore the table and lose a day of unrelated writes to save a copy step. And when you genuinely must restore, snapshot the current state first, under a name you will recognise at 03:00. Some configurations take such a failsafe automatically; take your own regardless, because it costs seconds and it is the only thing standing between a bad restore and a permanent one.

# capture (default mode flushes MemStore first)
snapshot 'orders', 'orders-20260805-0900'

# capture without forcing a flush - omits anything still in MemStore
snapshot 'orders', 'orders-fast', {SKIP_FLUSH => true}

list_snapshots

# safe: a new table backed by HFileLinks, no data copied
clone_snapshot 'orders-20260805-0900', 'orders_verify'

# destructive and in place: table must be disabled, later writes are lost
snapshot 'orders', 'orders-prerestore-20260806'   # take your own failsafe
disable 'orders'
restore_snapshot 'orders-20260805-0900'
enable 'orders'

# removes the manifest and the cleaner's veto; space returns on a later pass
delete_snapshot 'orders-fast'

ExportSnapshot for off-cluster backup, and why it is not replication

A snapshot lives in the same HDFS as the table it protects. That is fine for rollback and useless for disaster recovery: lose the cluster and you lose the snapshot with it. ExportSnapshot is the step that turns a snapshot into a backup. It is a MapReduce job that copies the manifest and every file the manifest names to another filesystem - another HDFS, or object storage.

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot \
  -snapshot orders-20260805-0900 \
  -copy-to hdfs://dr-cluster/hbase \
  -mappers 16

Unlike creation, this one really does move bytes: the transfer is sized by the snapshot's distinct file set, and -mappers is the parallelism knob you tune against available network and cluster capacity. The destination does not need a running HBase - it needs a filesystem. To use an exported snapshot you clone or restore it on a cluster whose HBase root is that destination.

The contrast with replication is point-in-time versus continuous, and it is the distinction that decides which one is a backup. Replication ships WAL edits as they are written, so the peer trails the source by seconds and converges on the same state - including converging on your mistakes, at the same speed as your data. A dropped column family propagates to the peer before anyone has read the alert. An exported snapshot is a state you deliberately chose at a moment you deliberately picked, which is exactly why it protects against logical damage and replication does not. Most serious deployments run both: replication for availability and read offload, exported snapshots for recovery from human and software error. For the scheduled full-plus-incremental scheme built on top of snapshots and WAL boundaries, see backup and restore.

Reading a snapshot directly from MapReduce or Spark

For many teams the use that actually justifies snapshots is not backup at all. A large batch scan through the normal client path goes RegionServer by RegionServer: RPC round trips, block cache churn that evicts the working set of your online reads, GC pressure, and a batch job competing with latency-sensitive traffic for the same heap. A snapshot lets you avoid all of it.

TableSnapshotInputFormat resolves the snapshot manifest into a set of store files, splits the work by region, and reads those files straight from HDFS. No RegionServer participates. The online cluster sees the job only as HDFS read traffic, which is a far more tractable thing to schedule around than heap pressure. It needs a scratch directory on HDFS in which to materialise the links it reads through, and it does not modify the snapshot itself - several jobs can read the same snapshot concurrently.

Two properties are worth knowing before you make this the default for analytics. First, splits map to regions, so parallelism is bounded by region count: a twelve-region table gives you twelve mappers no matter how many terabytes it holds. That is a concrete argument for not letting regions grow enormous, and for paying attention to region sizing and key distribution if batch throughput matters.

Second, and more often overlooked: reading files directly bypasses the RegionServer, and the RegionServer is where cell-level authorization and visibility labels are enforced. A job reading a snapshot is subject to filesystem permissions on the snapshot path and to nothing else. On a cluster where those controls are load-bearing, snapshot-based analytics is a hole unless the snapshot directories are permissioned deliberately. The read itself does go through the normal store-file scanner stack, so delete markers and version limits are honoured as in an ordinary scan - but the file set is frozen at snapshot time, so anything a later compaction would have physically purged is still present in the files being read.

Operational failure modes and retention policy

Snapshots nobody deletes. A default HBase configuration does not expire snapshots. A daily snapshot job with no matching reaper is one of the most common ways an HBase cluster quietly runs out of HDFS, and because the growth shows up under /hbase/archive it is usually diagnosed late. Retention has to be code you write: list the snapshots, parse the timestamp out of your own naming convention, delete anything past the window. Name them so a script can do that - something like <table>-<purpose>-<yyyymmddHHMM> - because the name and the creation time are essentially all the metadata a snapshot carries.

Deleting a snapshot does not free space immediately. delete_snapshot removes the manifest and lifts the cleaner's veto. The actual unlinking happens on a later cleaner pass, and the archive's time-to-live cleaner applies its own age threshold on top. Expect a lag between the delete and the capacity graph moving, and do not conclude from a flat graph that the delete failed.

Restore losing writes. This is the one that generates incidents, because "restore the snapshot" sounds additive and is not. Anything written after the snapshot is destroyed, and no WAL replay can bring it back from the snapshot. If the goal is recovering specific rows, clone and copy.

Wrong scope. A snapshot covers exactly one table. A workload spread across several tables gets no cross-table consistency from snapshotting them in a loop - each freezes at a different moment, and the skew is however long the loop took. If the application needs them to agree, you have to quiesce writes across all of them or document the skew as accepted.

Snapshot storms. Snapshotting many large tables at once in flush mode forces many simultaneous flushes and a wave of small files behind them, which lands as a compaction backlog minutes later. Stagger the schedule, and watch flush queue and compaction queue depth around the snapshot window rather than only at the moment the command runs - see alerting practice for which of those signals are worth paging on.

A note on the neighbouring feature. HDFS snapshots are a different mechanism at a different layer: they freeze a directory subtree at the filesystem level and are unaware of HBase's file lifecycle, MemStore, or table semantics. Taking an HDFS snapshot of /hbase gives you a filesystem-consistent image of files that may be mid-flush and mid-compaction, with in-memory writes missing and nothing coordinating region boundaries. Use HBase snapshots for HBase tables; the two are not interchangeable.