HBase has no delete thread. Nothing in a RegionServer wakes up on a timer, walks your table and removes data that has aged out. Retention is expressed as four attributes on a column family — TTL, VERSIONS, MIN_VERSIONS and KEEP_DELETED_CELLS — and those attributes are evaluated in exactly two places: by the scan matcher on every read, and by the merge that runs during a compaction. That split is the whole subject. A read gives you the right answer the instant you change a setting; the disk usage graph does not move until something rewrites the files, which may be days later or never. Almost every retention surprise in HBase — data that will not go away, data that vanished early, a deleted value that came back, a value that disappeared after a compaction — is that gap between the read path and the storage path.
The two axes: age and count
A value in HBase is addressed by four coordinates plus a type byte: row key, column family, column qualifier, and timestamp. The timestamp is not metadata attached to a value — it is part of the key. Two writes to the same row, family and qualifier at different timestamps are two distinct cells that coexist in the store, sorted newest first. That is all a version is.
Retention then has two independent knobs, and both live on the column family descriptor. TTL bounds age: a cell is dead once its timestamp is older than the current time minus the TTL. VERSIONS (the attribute the Java API calls MAX_VERSIONS) bounds count: only the newest N cells for a given row and qualifier survive. Neither is a quota on a row, a region or a table — both are applied per column, independently, everywhere in the family.
They compose as an intersection with a floor. A cell is kept if it is among the newest VERSIONS versions of its column and it has not aged past the TTL — except that MIN_VERSIONS rescues cells the TTL rule would have dropped. Because both settings are family-scoped, a row can be half expired: the family holding raw events ages out on a seven-day TTL while the family holding the derived summary, on the same row key, keeps its cells for a year. There is no row-level existence record in HBase, so when the last live cell of a row expires the row simply stops appearing in scans.
Where the settings live and what the defaults are
All four attributes are column family properties, set at create time or altered online. The shell spells them out; the Java API uses ColumnFamilyDescriptorBuilder.
# create with explicit retention
create 'events', {NAME => 'd', VERSIONS => 1, TTL => 604800}
# widen retention on a live table -- online, no data rewrite
alter 'events', {NAME => 'd', TTL => 2592000, VERSIONS => 3,
MIN_VERSIONS => 1, KEEP_DELETED_CELLS => 'TTL'}
describe 'events' # shows the effective values, TTL as seconds or FOREVERColumnFamilyDescriptor d = ColumnFamilyDescriptorBuilder
.newBuilder(Bytes.toBytes("d"))
.setTimeToLive(604800) // SECONDS
.setMaxVersions(3)
.setMinVersions(1)
.setKeepDeletedCells(KeepDeletedCells.TTL)
.build();| Attribute | Default | Unit and scope |
|---|---|---|
VERSIONS | 1 | count, per row and qualifier |
MIN_VERSIONS | 0 | count, per row and qualifier |
TTL | FOREVER | seconds, per cell timestamp |
KEEP_DELETED_CELLS | FALSE | enum: FALSE, TRUE, TTL |
Two default-related traps are worth naming. First, VERSIONS defaults to 1 in modern HBase; it defaulted to 3 in releases before 0.96, so tables created long ago and migrated forward may still carry a 3 that nobody chose. Run describe rather than assuming. Second, FOREVER is not a special case in the storage engine — it is Integer.MAX_VALUE seconds, about 68 years, and it is a real comparison the matcher performs on every cell.
An alter is a metadata change. The Master reopens the regions to pick up the new descriptor and the next read applies the new rules, but not one byte of data is rewritten by the alter itself. Tightening a TTL is therefore instant and free, and reclaims nothing.
Cell TTL: per mutation, in milliseconds, and it can only shorten
Since 0.98 a client can attach a TTL to an individual mutation, independent of the family TTL. It is set on the Mutation, so it applies to every cell in that Put, not to a named column.
Put p = new Put(rowKey);
p.addColumn(FAM, QUAL, value);
p.setTTL(3_600_000L); // MILLISECONDS -- one hourThe unit mismatch is the trap, and it is a factor of a thousand in the dangerous direction: the family TTL is in seconds and Mutation.setTTL is in milliseconds. Code that copies a configured value from one to the other without converting either expires data a thousand times too early or holds it a thousand times too long, and neither failure raises an error.
A cell TTL cannot extend a cell's life beyond the family TTL. It is a way to make some cells in a family die sooner than the family's cap, never a way to exempt a cell from it. Treating per-cell TTL as an override — the framing you will find in a lot of thin write-ups, including earlier notes on this site — gets this backwards and leads to a design where the important rows are expected to outlive a family TTL that will in fact remove them on schedule.
Mechanically the TTL rides along as a tag on the cell. Tags are only persisted by HFile version 3, which has been the default since HBase 1.0 but is worth confirming with hfile.format.version on an upgraded cluster, because on v2 files the tag is silently dropped and the cell quietly reverts to the family TTL. Tags are also not returned to ordinary clients, so there is no API that answers how long this cell has left — you recompute it from the cell timestamp, and the TTL you originally sent is not recoverable from the data. Tags are the same mechanism behind cell-level visibility labels and ACLs, so a family already using those is already paying the per-cell tag overhead.
MIN_VERSIONS: the floor that outranks expiry
MIN_VERSIONS is the least understood of the four, mostly because it does nothing at all unless a TTL is set. With no TTL nothing ever dies of old age, so there is no expiry for a floor to override, and the setting is inert.
With a TTL set, it means: keep at least this many of the newest versions of a column even if they are older than the TTL. The reason it exists is the last-known-value problem. A device-state table with a 30-day TTL and MIN_VERSIONS => 0 loses a sensor entirely if it stops reporting for a month — the row does not go stale, it disappears, and the application cannot distinguish never seen from silent for 31 days. With MIN_VERSIONS => 1 the most recent reading survives indefinitely and the application can read it, see how old the timestamp is, and decide what that means.
The cost is that the TTL is no longer a storage bound. Every column that ever received a write keeps at least MIN_VERSIONS cells forever, so the table's floor size is proportional to the number of distinct columns ever written, not to the active working set. On a table whose row keys are unbounded — one per session, one per request — that floor is the entire history and MIN_VERSIONS quietly converts a bounded table into an unbounded one. Use it on tables with a bounded key space (devices, accounts, feature keys) and avoid it on event streams.
MIN_VERSIONS must not exceed VERSIONS. And it is precisely the setting that KEEP_DELETED_CELLS => 'TTL' was designed to pair with, which is the subject of the next section.
Why expired data is invisible long before it is gone
When a RegionServer opens a scanner it builds a merged view over the MemStore and every HFile in the store that could contain the requested keys, and it feeds each cell through a scan query matcher that returns include, skip, or seek-ahead. The matcher computes an oldest-unexpired-timestamp bound once, from the server clock and the family TTL, and skips every cell below it; a separate column tracker counts versions per qualifier and stops emitting once it has produced VERSIONS of them.
So expired and superseded cells are never returned. There is no window during which a client can observe them through a normal get or scan, and no need to trigger anything to make a TTL change take effect on reads. Shorten a TTL from 30 days to 7 and the next scan behaves as though the older data is gone, on every region, immediately.
They are also, largely, not costing you read time. The store-file selection step skips any HFile whose maximum timestamp is already below the unexpired bound, so a file consisting entirely of expired cells is not opened at all — it is not merged, its blocks are not read, its bloom filter is not consulted. Expired data mostly costs disk, not latency. Where it does cost latency is the opposite case: a file that straddles the boundary is read normally, and the matcher pays to skip cell by cell.
The corollary is the single most common support question about HBase retention. You tightened the TTL, reads confirm the data is gone, and hdfs dfs -du on the table directory has not moved. Nothing is broken. The read path and the storage path answer to different clocks, and only the second one is tied to compaction.
Deletes, tombstones, and KEEP_DELETED_CELLS
Start by separating two things that thin write-ups routinely conflate. TTL expiry writes nothing. No tombstone, no marker, no I/O, no RPC — a cell becomes expired purely because the wall clock moved past its timestamp plus the TTL. An explicit delete is the opposite: it is a write, it makes the store bigger, and what it writes is a marker cell.
There are four marker types, and confusing them is a real source of bugs. A version delete removes one exact timestamp. A column delete removes every version of one qualifier at or below a timestamp. A family delete removes every qualifier in the family at or below a timestamp. A family-version delete removes every qualifier in the family at one exact timestamp. All of them are bounded by a timestamp, and that is what produces the classic rule that deletes mask puts: a marker at time T hides matching data at or below T, including data written later that carries an older timestamp. A backfill job that replays historical events into a range someone has deleted writes cells that are invisible from the moment they land.
KEEP_DELETED_CELLS decides how long the shadowed data stays retrievable. FALSE, the default, means deleted cells are not retained — a time-range scan that ends before the delete still will not see them. TRUE retains deleted cells until they are removed by some other means, such as the version limit or the TTL, which with no TTL and no further writes means forever. TTL retains them only until the delete marker itself ages out of the family TTL, which is the setting to use when you have combined a TTL with MIN_VERSIONS and want deletes to actually take effect eventually.
To see any of this you need a raw scan, which bypasses the matcher's delete tracking and returns markers and shadowed cells as they exist on disk. Note the restriction: it is an error to name columns in a raw scan.
scan 'events', {RAW => true, VERSIONS => 10, LIMIT => 5}
# type=Delete / DeleteColumn / DeleteFamily rows are the markersWhich compaction actually frees the bytes
A compaction merges sorted store files into new ones, and it runs the same matcher the read path runs. Reclamation is not a separate step: the merge simply does not write out the cells the matcher declares dead, and the input files are then deleted. That is the only mechanism by which HBase gets smaller.
The distinction that matters is what the compaction can see. A minor compaction merges a subset of the store's files, so it cannot prove that some file it did not include holds data an older delete marker must keep masking. It therefore carries markers and the cells they shadow through into its output. A major compaction merges every file in the store into one, which makes that proof trivial — and so it is the only run that can drop delete markers and the data behind them. Plan your capacity on the documented guarantee: space comes back at major compaction.
There is one cheap exception worth knowing, because on time-series tables it does most of the work. If every cell in a store file is expired, the file can be dropped whole during ordinary minor compaction without being rewritten at all — controlled by hbase.store.delete.expired.storefile, on by default. This is also the strongest argument for date-tiered compaction on TTL-heavy tables: it groups cells into files by timestamp window, which maximises the number of files that are entirely expired and can be dropped for free instead of rewritten. See HBase compaction for the policy machinery and the HFile format for the per-file timestamp range that makes the whole-file drop possible.
Scheduling is the operational half. hbase.hregion.majorcompaction defaults to seven days with a jitter factor, so regions spread themselves over a wide window rather than storming at once. Many production clusters set it to 0 and issue major_compact from a scheduler during a quiet window instead, because a major compaction rewrites the entire store and competes directly with serving I/O. If you do that and then forget the scheduler, retention stops working — reads stay correct, and the table grows forever.
The client owns the timestamp, so the client owns expiry
TTL is evaluated against the cell timestamp, not against when the write arrived. The timestamp defaults to the RegionServer's clock, but any client can supply its own, and that makes expiry a property the application controls whether or not it realises it.
The most common consequence is data that is born expired. A backfill that stamps each cell with the event's original time, loaded into a family with a 30-day TTL, will not return anything older than 30 days — the load succeeds, the counters look right, the scan comes back empty, and the next compaction quietly removes it. The same applies to bulk-loaded HFiles, which carry their original timestamps into the region unchanged; see HBase bulk load. If you need event time as a query dimension, put it in the row key, and let the cell timestamp mean write time.
Unit bugs bite the same way. A client that writes seconds where HBase expects milliseconds stamps everything in early 1970, so every cell is expired on arrival under any TTL. The reverse — milliseconds where the intent was seconds, or a deliberate far-future timestamp — produces cells that never expire and, worse, that mask every subsequent write to the same column, because a newer write with a real timestamp sorts below them. Clock skew between RegionServers shifts the boundary in both directions across a table, which is one more reason clusters run NTP.
Finally, writing repeatedly at the same timestamp is not versioning at all. Same row, family, qualifier and timestamp is the same cell coordinate; the values coexist in different files until a compaction resolves them, and which one survives is not something to rely on. If you want N distinct versions, write N distinct timestamps. And resist the urge to set VERSIONS to hundreds to build an audit trail — every version is a full cell with a full key, and store file size grows accordingly.
The operational surprises, in the order teams hit them
Tightening a TTL does not free space. Covered above and worth repeating because it is usually discovered during an incident: reads are correct immediately, disk is unchanged until a compaction rewrites the files. If you need the space now, run major_compact on the table and accept the I/O.
Deleting makes the table bigger first. A bulk delete writes a marker per affected cell or column, and until a major compaction runs you are storing the original data, the markers, and paying to skip both on every read. Delete-heavy workloads need a major compaction cadence sized to the delete rate, not the default.
A major compaction can change query results. This is documented HBase behaviour and it startles people: a time-range get that returned an old version can stop returning it after a compaction, because that version was already beyond VERSIONS and was only still readable in a file that had not yet been rewritten. Before compaction, VERSIONS is a guarantee of a minimum, not a maximum. Never build application logic on versions the descriptor does not promise to keep.
Snapshots pin expired data. When a compaction retires a file that a snapshot still references, the file is moved to the archive directory rather than deleted, and cluster disk usage does not drop. A table under a weekly snapshot schedule with a daily TTL can hold far more than the TTL implies; the space returns when the snapshots expire. See HBase snapshots.
Retention does not replicate. Column family descriptors are per cluster. A replication peer with a longer TTL keeps data the source has dropped, one with a shorter TTL loses data the source still serves, and mismatched KEEP_DELETED_CELLS makes the two clusters answer time-range queries differently. Deletes propagate as markers and are subject to the peer's own rules once they land. See HBase replication.
Raising VERSIONS does not restore history. Versions already discarded by a compaction are gone. The new setting applies from the moment the descriptor changes, forward only.
Choosing the four settings per workload
Retention design is four decisions, and the useful discipline is to make them explicitly rather than inheriting defaults. The table below is the set of shapes that covers most real tables.
| Workload | VERSIONS | MIN_VERSIONS | TTL | KEEP_DELETED_CELLS |
|---|---|---|---|---|
| Event or log stream | 1 | 0 | days to weeks | FALSE |
| Last-known device or account state | 1 to 3 | 1 | weeks | TTL |
| Audit or slowly changing dimension | 10 or more | 0 | years | TRUE |
| Session or cache data | 1 | 0 | minutes to hours | FALSE |
| Reference or lookup data | 1 | 0 | FOREVER | FALSE |
Three rules of thumb sit behind that table. Split families by retention, not by convenience: because TTL and VERSIONS are family-scoped, a column that needs a different retention needs a different family, and pushing everything into one family forces you to retain everything at the longest requirement. The counter-pressure is that families are not free — each one is a separate store with its own MemStore and its own files, and flushes are triggered across all families of a region — so a handful of families with clearly different retention is right and a dozen is not.
Second, set VERSIONS to 1 unless you have a named consumer of history. Version retention is usually inherited rather than chosen, and it costs storage on every update. Third, prefer a TTL over a delete job wherever the retention rule is expressible as an age. A TTL costs nothing at write time and nothing at read time beyond a timestamp comparison; a scan-and-delete job writes markers, doubles the data until the next major compaction, and competes with serving traffic the whole way.
If your retention rule genuinely is not expressible as age or count — retain while a row is referenced elsewhere, keep the first and last version of each day — the extension point is a RegionObserver coprocessor that wraps the compaction scanner and filters cells as they are merged. That is the supported way to add a custom retention policy without a delete job.
Verifying that reclamation actually happened
Because reads hide expired data, reads cannot tell you whether retention is working. Three independent measurements can.
# 1. what the descriptor actually says, per family
echo "describe 'events'" | hbase shell -n
# 2. bytes on disk for the table, and what snapshots are pinning
hdfs dfs -du -h /hbase/data/default/events
hdfs dfs -du -h /hbase/archive/data/default/events
# 3. what a store file really holds: timestamp range and cell count
hbase hfile -m -f /hbase/data/default/events/<region>/d/<file>
# force reclamation and re-measure
echo "major_compact 'events'" | hbase shell -nThe HFile metadata dump is the decisive one. It prints the file's timestamp range, so you can see directly whether a file predates the TTL horizon in its entirety — the case that gets dropped whole on a minor compaction — or straddles it and will need a rewrite. Add the key-value print flag on a small file to see delete markers and multiple versions exactly as they are stored, which is the ground truth a raw scan approximates.
Comparing the data directory against the archive directory answers the other half. If the table directory shrank after a major compaction and total HDFS usage did not, the bytes moved to the archive and something — almost always a snapshot, occasionally a replication peer that has fallen behind — is still holding a reference. See HBase snapshots for the reference-counting rules.
The routine that keeps this healthy is short: describe every family and record its retention next to the owning team; alert on table size against the size the TTL implies rather than against a fixed threshold, since the ratio between them is the reclamation lag; watch the compaction queue and the age of the last major compaction per region, because a stalled queue is a stalled retention policy; and re-check after any restore or bulk load, both of which can reintroduce data with timestamps that make it either instantly expired or effectively immortal. Operational coverage of the compaction queue itself lives in the RegionServer article.