Why architecture matters here
Architecture here matters because compaction quietly changes the contract of a topic, and every consumer must be written to that contract. On a delete-retention topic, a consumer that starts from the earliest offset sees the complete history; on a compacted topic it sees a compressed history where intermediate values may be gone. If your consumer computes something that depends on seeing every update — a running count of address changes, say — compaction silently corrupts it. The right mental model is: a compacted topic promises the latest value per key and the ordering of what remains, but not the presence of any particular older value. Design consumers to be idempotent and last-write-wins, and compaction is a gift; design them to count events, and it is a bug.
The economic argument is just as important. State topics grow with write volume but the useful information grows only with key cardinality. A user-preferences topic for ten million users might absorb a billion writes a month, yet the current state is ten million records. Without compaction you pay to store and replay the billion; with it you pay for the ten million. For any system that bootstraps state by replay — Kafka Streams restoring a store, a cache warming on deploy, a search index rebuilding — that difference is the gap between a thirty-second restart and a thirty-minute one.
Finally, compaction is what lets Kafka be a system of record for state rather than merely a transport. Because the latest value survives indefinitely, the log itself is the source of truth; downstream stores are just materialized views you can throw away and rebuild. That inversion — the log is primary, the database is a cache — is the architectural idea that makes event-sourced and CDC-driven systems robust, and it only holds because compaction guarantees the head of every key is never lost.
There is also a subtle but important reliability dividend: because the compacted log carries the full current state, it doubles as a disaster-recovery artifact. If a downstream store is lost or corrupted, you do not restore a database backup and replay a week of events on top of it — you simply replay the compacted topic from offset zero and the store rebuilds itself to the exact current state, bounded by key count. That property turns a whole class of 'restore the database' incidents into routine, well-understood replays, and it is why compaction is not just a space optimization but a foundational piece of how these systems stay recoverable.
The architecture: every piece explained
A partition's log is a sequence of segments, each a file of records. The newest segment is the active segment — the one currently being written — and the cleaner never touches it, so the most recent writes are always safe. Everything behind the active segment is divided into the clean portion (already compacted: at most one record per key) and the dirty portion (written since the last compaction, possibly many records per key). The boundary between them is the cleaner point, an offset the broker tracks per partition.
The log cleaner is a pool of background threads. When it selects a partition, it makes two passes. First it scans the dirty section and builds an offset map: a compact hash table from key to the highest offset at which that key appears. Then it recopies the log — clean plus dirty — into new segments, writing a record only if its offset equals the map's entry for its key. Anything older for the same key is dropped. The rewritten segments replace the originals atomically, and the cleaner point advances. The offset map is memory-bounded (log.cleaner.dedupe.buffer.size), which caps how many distinct keys one pass can handle; very high-cardinality partitions may need multiple passes.
Deletion uses tombstones: a record with the key set and the value set to null. During compaction a tombstone supersedes all earlier values for its key, exactly like any newer value — but the tombstone itself must also eventually be removed, or the log could never shrink. Kafka keeps tombstones for delete.retention.ms after they reach the clean section, then drops them. Two knobs govern how eagerly the cleaner runs: min.cleanable.dirty.ratio (the fraction of the log that must be dirty before a partition is eligible) and min.compaction.lag.ms / max.compaction.lag.ms (the minimum age before a record can be compacted, and the maximum before it must be). Together they trade CPU and IO against how quickly superseded values disappear.
One more subtlety governs how much the cleaner can reclaim in a single pass: the offset map is sized by log.cleaner.dedupe.buffer.size divided across log.cleaner.threads, and it must hold an entry for every distinct key in the dirty region a thread processes at once. If a partition's dirty section contains more unique keys than the buffer can index, the cleaner processes it in multiple rounds, each reclaiming only part of the redundancy, so high-cardinality partitions compact more slowly and stay larger than their theoretical floor. This is why sizing the dedupe buffer against your largest partition's key count is not a micro-optimization but a correctness-of-space decision — an undersized buffer silently leaves duplicate values on disk that you assumed compaction had already removed.
End-to-end flow
Follow a single key. A producer writes userId=42 → {city: Austin} at offset 100, later userId=42 → {city: Denver} at offset 5000, both landing first in the active segment. The head of the log now has two records for key 42; a consumer replaying from earliest would see Austin then Denver, and a last-write-wins consumer correctly ends on Denver. Nothing is compacted yet — the writes are still in or near the active segment.
Time passes and those segments roll out of the active position into the dirty section. Eventually the dirty ratio for the partition crosses min.cleanable.dirty.ratio, the cleaner picks it up, scans the dirty region, and records 42 → 5000 in its offset map (the highest offset for key 42). On the recopy pass, the offset-100 Austin record is skipped because 100 ≠ 5000; the offset-5000 Denver record is retained. The rewritten clean segment now holds one record for key 42, and the cleaner point advances past offset 5000. A fresh consumer starting from earliest now sees only Denver for key 42 — the snapshot view.
Now delete the user: the producer writes userId=42 → null (a tombstone) at offset 9000. It lives in the head like any write. When compaction next runs, the tombstone supersedes Denver, so the Denver record is dropped and only the tombstone remains — a consumer replaying still learns 'key 42 was deleted.' After the tombstone has spent delete.retention.ms in the clean section, the cleaner removes it too, and key 42 vanishes from the log entirely. A Kafka Streams application consuming this topic would, in order: put Austin, overwrite with Denver, then delete on the tombstone — arriving at exactly the current state regardless of when it started, as long as it started before the tombstone aged out.
It is worth noticing what a consumer that joined before any of this sees versus one that joins after. A consumer already attached at offset 0 when the Austin write landed observes the full changelog — Austin, then Denver, then the tombstone — and applies each in turn, because it read the records before the cleaner removed them. A brand-new consumer that starts from earliest only after compaction sees the compressed history: perhaps just Denver, or just the tombstone, depending on timing. Both arrive at the same final state, which is exactly the guarantee compaction makes — the destination is preserved even though the path is not — and it is why every consumer of a compacted topic must be written as an idempotent, last-write-wins state machine rather than an event counter.