Why architecture matters here
Compaction matters because it's essential to LSM read performance and space reclamation, but it's also a major source of I/O cost and latency spikes -- so managing it correctly is central to HBase performance. The LSM write path (fast writes, accumulating HFiles) has two problems that compaction solves: read amplification (reads must check all the HFiles -- growing with the HFile count, so reads slow as HFiles accumulate -- compaction merging HFiles to bound the count) and space (deleted/expired data lingering -- reclaimed only by major compaction). So compaction is essential (without it, reads slow unboundedly and space grows). But compaction is costly (write amplification -- rewriting data; compaction I/O -- competing with serving, causing latency spikes). So compaction is a double-edged necessity (essential for read performance and space, but costly in I/O and latency) -- and managing it (scheduling, throttling, tuning the policy) is central to HBase performance (balancing the read/space benefit against the I/O/latency cost). For operating HBase (or any LSM store -- Cassandra, RocksDB), understanding compaction (why it's needed, its cost, how to manage it) is essential, and mismanaging it (too little -- read amplification; too much or ill-timed -- I/O/latency) is a common HBase performance problem.
The read-amplification-vs-cost insight is the architectural crux, and it's the fundamental LSM tradeoff compaction manages. The LSM structure inherently trades off three amplifications. Read amplification: a read may check multiple HFiles (the more HFiles, the more a read checks -- higher read amplification, slower reads). Write amplification: data is rewritten multiple times (each compaction rewrites the merged data -- so a piece of data is written multiple times over its life -- write amplification). Space amplification: obsolete data (superseded versions, deleted/expired data not yet reclaimed) takes space (until compaction reclaims it -- space amplification). Compaction manages these tradeoffs: more compaction (merging aggressively) reduces read amplification (fewer HFiles) and space amplification (reclaiming obsolete data) but increases write amplification (more rewriting) and I/O (the compaction work). Less compaction (merging lazily) reduces write amplification and I/O but increases read amplification (more HFiles) and space amplification (obsolete data lingering). So the compaction policy tunes this tradeoff (how aggressively to compact -- balancing read amplification and space against write amplification and I/O). Understanding this fundamental LSM tradeoff (read vs write vs space amplification, managed by the compaction policy) is understanding what compaction fundamentally does and how to tune it.
And the tombstone-reclamation-on-major-compaction reality is a crucial, often-surprising detail. In an LSM store, a delete doesn't immediately remove the data -- it writes a tombstone (a marker that the row is deleted -- since the data is in immutable HFiles that can't be modified in place). The actual data (the deleted row, and its tombstone) remains in the HFiles until a compaction reclaims it. And crucially, tombstones (and the data they shadow) are fully reclaimed only during a major compaction (which merges all the HFiles for a region -- so it can see all the versions and the tombstone, and drop them; a minor compaction, merging only some HFiles, can't safely drop a tombstone if older data might be in a non-merged HFile). Similarly, TTL-expired data is reclaimed on compaction (major compaction fully reclaiming it). This has important consequences: deleted/expired data lingers (taking space, and even affecting reads -- a read seeing the tombstone) until a major compaction reclaims it -- so if major compactions are infrequent (or disabled), deleted data accumulates (space, and read overhead from tombstones -- a common issue with delete-heavy workloads -- the tombstones accumulating between major compactions). Understanding that tombstones and deleted/expired data are reclaimed only on major compaction (so deletes don't immediately free space, and tombstones accumulate until major compaction) is understanding a crucial, often-surprising aspect of LSM deletes and why major compaction scheduling matters (especially for delete-heavy workloads).
The architecture: every piece explained
Top row: the write path and compaction types. LSM writes: writes go to the memstore (in-memory) and WAL (durability), and flush to immutable HFiles when the memstore fills -- fast writes (sequential flushes), accumulating HFiles. Many HFiles: HFiles accumulate over time, and reads must check all HFiles that might contain the row -- so read amplification grows with the HFile count (more HFiles = slower reads). Minor compaction: merges some HFiles (a subset -- reducing the HFile count, bounding read amplification) -- frequent, smaller compactions. Major compaction: merges all the HFiles for a region into one (and drops tombstones and TTL-expired data -- reclaiming space) -- less frequent, larger, and the only compaction that fully reclaims deleted/expired data.
Middle row: reclamation and cost. Tombstones and TTL: deletes write tombstones (markers -- the data in immutable HFiles not removed in place), and TTL-expired data lingers -- both reclaimed on major compaction (which sees all versions and drops them). Write amplification: data is rewritten repeatedly (each compaction rewrites the merged data -- so data is written multiple times over its life) -- the write cost of compaction. Compaction I/O: the merging reads and writes a lot of data (competing with the serving I/O -- potentially causing latency spikes during compaction) -- the I/O cost. Scheduling and throttling: managing the compaction I/O -- scheduling major compactions off-peak (when serving load is low), throttling the compaction I/O rate (limiting its competition with serving) -- controlling the compaction cost's impact.
Bottom rows: policy and tradeoff. Compaction policy: which HFiles to compact and when (the policy -- e.g., size-tiered: compacting HFiles of similar size; or by count/age -- selecting the files and timing) -- tuning the compaction behavior. Read vs write vs space: the fundamental LSM tradeoff -- compaction trades write amplification and I/O (the cost) for read amplification (bounding HFiles) and space (reclaiming obsolete data) -- the policy tuning this balance (more compaction = better reads/space, worse write/I/O; less = the reverse). The ops strip: tuning (tuning the compaction policy and parameters -- how aggressively to compact, the file selection -- for the workload's read/write/space balance), scheduling (scheduling major compactions -- off-peak, and frequency -- balancing reclamation against the I/O impact; especially for delete-heavy workloads needing regular major compactions), and monitoring (monitoring the HFile counts -- read amplification; the compaction activity -- I/O; the tombstone/space -- reclamation -- to tune the compaction).
End-to-end flow
Trace compaction managing read amplification. A region receives steady writes: the memstore flushes to HFiles periodically, accumulating HFiles over time. As the HFiles accumulate (say, dozens), reads slow (a read must check all the HFiles that might contain the row -- more HFiles = more checking = higher read amplification, slower reads). Minor compactions kick in (merging subsets of HFiles -- reducing the count -- e.g., merging several small HFiles into one larger one) -- bounding the HFile count (keeping read amplification in check -- reads checking fewer HFiles). Periodically, a major compaction runs (merging all the region's HFiles into one -- so a read checks just one HFile -- minimal read amplification -- and dropping tombstones/expired data -- reclaiming space). The compaction (minor bounding the count, major consolidating and reclaiming) manages the read amplification (keeping reads fast despite the steady writes accumulating HFiles) -- the essential role of compaction (bounding read amplification for the LSM write path).
The tombstone and I/O-cost vignettes show the reclamation and cost. A tombstone case: a delete-heavy workload writes many deletes (tombstones -- markers, the data not immediately removed). Between major compactions, the tombstones (and the shadowed deleted data) accumulate (taking space, and adding read overhead -- reads seeing the tombstones) -- and the space isn't reclaimed (the deletes not actually freeing space until a major compaction). The team ensures regular major compactions (which drop the tombstones and reclaim the deleted data's space) -- so for the delete-heavy workload, regular major compactions are important (reclaiming the deleted data's space and clearing the accumulated tombstones) -- illustrating that deletes are reclaimed only on major compaction. An I/O-cost case: a major compaction (merging all the region's HFiles -- a lot of I/O) runs during peak serving hours -- causing a latency spike (the compaction I/O competing with the serving I/O). The team addresses it: scheduling major compactions off-peak (when serving load is low -- avoiding the competition) and throttling the compaction I/O rate (limiting its impact on serving) -- managing the compaction I/O cost's impact on serving latency.
The policy and tradeoff vignettes complete it. A policy case: the team tunes the compaction policy for their workload -- a read-heavy workload benefits from more aggressive compaction (fewer HFiles -- lower read amplification -- faster reads, accepting the higher write amplification/I/O), while a write-heavy workload might tune for less aggressive compaction (less write amplification/I/O, accepting somewhat higher read amplification) -- tuning the policy for the read/write balance. A tradeoff case: the team recognizes the fundamental LSM tradeoff (read vs write vs space amplification) -- and tunes the compaction (aggressiveness, scheduling) to balance it for their workload and constraints (read latency requirements, write throughput, space budget, serving-latency sensitivity) -- the compaction tuning being the LSM tradeoff management. The consolidated discipline the team documents: understand compaction as essential LSM management (bounding read amplification -- merging HFiles; reclaiming space -- major compaction dropping tombstones/expired data), recognize the fundamental LSM tradeoff (read vs write vs space amplification -- tuned by the compaction policy), ensure regular major compactions (especially for delete-heavy workloads -- reclaiming deleted data, clearing tombstones), manage the compaction I/O cost (scheduling off-peak, throttling -- avoiding serving-latency spikes), tune the compaction policy for the workload (read/write/space balance), and monitor (HFile counts -- read amplification; compaction I/O; tombstone/space -- reclamation) -- because compaction is essential to LSM read performance and space reclamation but costly in I/O and latency, and managing it (the read/write/space tradeoff, scheduling, throttling) is central to HBase performance.