Why architecture matters here
The architecture matters because the delta model trades write simplicity for read amplification, and compaction is the only thing that pays that debt back. On an immutable filesystem you cannot edit a row, so an update is expressed as a new version in a delta plus, effectively, a tombstone for the old one. That is a clean way to get transactional semantics, but it means the physical layout drifts steadily away from the logical state: the truth of the table is scattered across an ever-growing set of files that a reader must reassemble every single time. Compaction periodically collapses that scatter back into a compact representation.
The second reason is the small-files problem, which in Hadoop is not a nuisance but a systemic hazard. Each file consumes NameNode memory for its metadata and costs a task setup, an open, and a seek at read time. Streaming ingestion into an ACID table can manufacture thousands of tiny deltas per hour; multiply across partitions and tables and the NameNode's object budget and the cluster's task overhead are both consumed by files that hold almost no data. Compaction is the mechanism that keeps the file count bounded so the storage layer stays healthy.
The third reason is that reads and space both degrade continuously without it, so there is no stable operating point absent compaction. Query latency grows roughly with the number of deltas because the merge cost is proportional to the files touched; meanwhile deleted and updated rows are never physically removed — the old values persist in the base until a major compaction rewrites it — so disk usage inflates with churn. A table under steady update pressure gets both slower and larger over time, and only compaction reverses both curves.
The fourth reason is that this maintenance must be invisible to workloads. A warehouse cannot take a lock and rewrite a petabyte-scale table during business hours, and it cannot make writers wait while it tidies up. The compaction architecture is built around write-ids and snapshot isolation precisely so that a compaction job reads a consistent set of files, writes new ones, and atomically swaps them in — while concurrent readers continue against the old files and writers continue appending new deltas, none of them blocked. The safety of running maintenance concurrently with traffic is the whole reason it can be automatic.
Finally, the architecture matters because the choice of minor versus major, and how aggressively to schedule each, is a real cost-latency tradeoff the operator controls. Minor compactions are cheap and frequent and keep the file count in check but leave deletes unapplied; major compactions are expensive and less frequent but produce the cleanest layout and actually reclaim space from churn. Understanding the pipeline is what lets a team tune these thresholds so tables stay fast without spending more cluster capacity on compaction than on queries.
The architecture: every piece explained
Start with the on-disk shapes. A partition of an ACID table contains a base directory (base_N, where N is a write-id high-water mark) representing fully-compacted state, plus a set of delta directories (delta_x_y) holding inserted/updated rows and delete-delta directories (delete_delta_x_y) holding tombstones, each labeled by the write-id range it covers. A row's current value is the base as amended by deltas, with any row named in a delete-delta suppressed.
The Initiator is the Metastore thread that decides what to compact. On a schedule it examines each table and partition and applies thresholds: if the number of delta directories, or the ratio of delta size to base size, crosses a configured limit, it enqueues a compaction request — choosing minor when the issue is too many deltas and major when the deltas are large relative to the base or deletes have accumulated enough that a full rewrite is warranted. The Initiator only decides and enqueues; it does no data movement itself.
The Worker threads consume the compaction queue and run the actual jobs. A worker launches a MapReduce or Tez job that reads the relevant files and writes the compacted output. A minor compaction reads the delta and delete-delta directories and merges them into a single consolidated delta set, dramatically cutting the file count while leaving the base untouched and deletes still expressed as tombstones. A major compaction reads the base plus all deltas, applies every insert, update, and delete, and writes a fresh base_N that is the definitive current state — at which point the deletes are physically gone and space from churn is reclaimed.
The Cleaner closes the loop. A compaction does not delete the old files immediately, because in-flight queries started before the compaction committed may still be reading them. Instead, the new compacted files are made visible, and the Cleaner waits until no active reader's valid-write-id snapshot still requires the superseded deltas and base; only then does it delete them, reclaiming the space. This deferred cleanup, gated on the lowest active reader's write-id, is what makes compaction safe to run concurrently with queries.
Tying it together is the write-id and valid-write-id list abstraction. Every transaction gets a write-id; every reader is given a snapshot list of which write-ids are valid (committed and visible) at query start. Readers merge exactly the files whose write-ids are valid for them, ignoring compaction output that committed after they began and ignoring aborted transactions' deltas entirely. Because compaction produces new files with their own write-id metadata and never mutates existing ones, a reader always sees a consistent view whether or not a compaction is running underneath it. The diagram shows the writers, the base+delta layout, the Initiator/Worker/Cleaner pipeline, and the readers merging at query time.
End-to-end flow
Trace a streaming table. A Kafka-to-Hive ingestion job writes micro-batches into the events table every few seconds; each commit creates a small delta_x_x directory with its own write-id. Simultaneously a nightly reconciliation job issues updates and deletes, which land as additional deltas and delete-deltas. Within an hour the active partition holds a base plus several hundred delta directories.
Queries against the partition during this hour must, for correctness, open the base and merge every delta and delete-delta whose write-id is valid in their snapshot. As the delta count climbs, each query's planning and I/O grow: more files to list, open, and merge. A dashboard that scanned the partition in two seconds when it was freshly compacted now takes fifteen, purely because of the file count. Nothing is wrong logically — the answer is correct — but read amplification is eating the latency budget.
The Initiator's periodic check notices that the partition's delta count has crossed the minor-compaction threshold. It enqueues a minor compaction request in the Metastore's compaction queue. A free Worker picks it up and launches a Tez job that reads the several hundred deltas and delete-deltas and writes them out as a single consolidated delta set with a write-id range spanning the merged transactions. When the job commits, the consolidated delta becomes visible to new readers, and the partition now presents the base plus one delta instead of hundreds.
New queries immediately benefit — they open a base and one delta and finish in a couple of seconds again. Queries that were already in flight against the old layout continue reading the original hundreds of deltas against their own snapshot, unaffected and correct. Once those readers finish, the Cleaner observes that no active valid-write-id list still references the superseded deltas and deletes them, reclaiming the space they occupied. The file count is back under control with zero disruption to traffic.
Over the day, churn from the reconciliation job accumulates updated and deleted rows whose old values still live in the base. When the ratio of delta content to base size crosses the major-compaction threshold — or on a scheduled cadence — the Initiator enqueues a major compaction. A Worker rewrites the base plus all deltas into a fresh base_N that fully applies every update and delete; the deleted rows physically vanish, disk usage drops, and the partition is now a single clean base with no deltas. The cycle then begins again as new writes arrive. This interplay — frequent cheap minors to bound file count, periodic expensive majors to reclaim space and apply deletes — is the steady state that keeps an ACID table fast and compact.