Why architecture matters here
LSM compaction matters because LSM databases live and die by their compaction quality. Too little compaction and reads slow down as they must merge across many SSTables. Too much and writes stall because the compaction thread is stealing IO from the write path. The wrong strategy for your workload turns a good database into a bad one.
Cost is real. Write amplification — total bytes written to disk divided by user bytes written — is often 10x or more in LSMs. Storage IO and SSD wear scale with that number. Right strategy cuts this dramatically.
Reliability under pressure is where compaction tuning shows. Under heavy write load, compaction can fall behind; falling further and further behind creates a runaway condition. Throttling, priorities, and disk IO isolation prevent this.
The architecture: every strategy explained
Walk the diagram top to bottom.
Writes. Every write appends to a Write-Ahead Log (WAL) for durability and inserts into an in-memory memtable. The WAL survives crashes; the memtable serves reads while it lives.
Memtable Flush. When the memtable reaches a threshold (typically 128-512 MB), it's flushed to disk as a sorted, immutable SSTable file. The memtable resets; writes continue to a new one.
Level 0 SSTables. Freshly flushed SSTables. In some strategies, L0 SSTables can have overlapping key ranges (many SSTables can contain the same key). Reads must consult all of them.
STCS (Size-Tiered). Cassandra's default and RocksDB's option. Group SSTables by similar size; when a tier has enough SSTables, merge them into one larger SSTable in the next tier. Great write amplification (each row written to disk ~log2(N) times). Terrible space amplification during compaction.
LCS (Leveled). LevelDB, RocksDB default, and Cassandra alternative. Each level is size-bounded (L1=10x L0, L2=10x L1). SSTables within a level have non-overlapping ranges. Reads consult at most one SSTable per level. Low read amp, low space amp, high write amp (~10x STCS).
TWCS (Time-Window). Cassandra strategy for time-series. Partition SSTables into time windows; compact only within a window; drop entire windows on expiry. Ideal for TTL workloads.
Compaction Trigger. Different for each strategy. STCS: N similar-sized SSTables in a tier. LCS: level over size threshold. TWCS: window rolled or old window compacts fully.
Throttle + Pace. IO budget per compaction thread; priority for user reads vs background compaction; rate limits to prevent runaway.
Read Amp vs Write Amp. The core trade. Read amp = SSTables touched per read; write amp = disk writes per user write. STCS: low WA, high RA. LCS: high WA, low RA. TWCS: workload-specific.
Space Amp. Duplicates on disk before compaction removes them. STCS peaks at 2x during major compaction; LCS stays near 1.1x.
End-to-end compaction flow
Trace a Cassandra workload with STCS. Writes stream in at 50k/sec. Memtable fills in 60 seconds; flushes to L0 SSTable of ~500 MB.
Over 30 minutes, 30 SSTables accumulate at L0. STCS sees 4+ SSTables of similar size and triggers minor compaction: merge them into one ~2 GB SSTable in the next tier. The 4 old SSTables are deleted (once no reads reference them).
Reads happen concurrently. A row lookup consults the memtable, the bloom filter for each L0 SSTable, and any tier SSTables. Bloom filters skip SSTables that definitely don't contain the row. Actual disk reads: 1-3 typical.
Major compaction: once weekly or on-demand, all SSTables merge into one huge one. Removes all tombstones. Frees a lot of space. Slow, IO-heavy — schedule off-peak.
Now switch to LCS on the same workload. L0 fills fast; LCS eagerly moves SSTables to L1. Each promotion requires rewriting overlapping L1 SSTables to keep the level non-overlapping. Write amp is 10x. But reads consult only ~5 SSTables total (one per level). Read latency drops.
Compare TWCS on a time-series workload. Data has a 7-day TTL. TWCS creates one SSTable per hour. After 7 days, entire SSTables drop; no compaction needed for expiration. Ideal for the pattern.