Why architecture matters here
The write path's design answers one question under hard constraints: how do you accept millions of random-key writes per second on hardware where random IO is ruinous, without losing a single acknowledged edit? The LSM answer decomposes it. Durability comes from the WAL: one sequential append stream per RegionServer (multiwal: a few), fsync'd to the HDFS pipeline before acknowledgment — random writes made durable at sequential-write prices. Order comes from the MemStore: edits land in a concurrent skiplist already sorted by rowkey, so when a flush writes an HFile the file is born sorted — no sort phase, and readers get binary-searchable files forever after. Amortization comes from batching in memory: a flush writes megabytes at once, and compaction later merges files at sequential speeds. Every mature write-optimized store — Cassandra, RocksDB, LevelDB — plays this same trio; HBase is where many engineers first meet it wearing production scars.
Why the details matter operationally: the coupling between stages is where incidents breed. WAL sync latency sits directly on every write's critical path — a slow HDFS datanode in the pipeline becomes a cluster-visible p99 spike. Flush pace couples to WAL retention: logs archive only when their edits are all flushed, so a cold column family that rarely flushes pins WAL files until forced flushes fire — the classic 'why do we have 900 WAL files' page. Flush pace also couples to compaction: flush too often (small MemStores) and you mint tiny HFiles that compaction must endlessly re-merge; too rarely (giant MemStores) and GC pressure and recovery-replay times balloon. The blockers convert these imbalances into client-visible pauses precisely so operators see them — reading blocker metrics is reading the system's stress honestly.
And recovery semantics decide real durability: acknowledged writes survive a RegionServer death because — and only because — WAL split and replay reconstruct un-flushed MemStores on the regions' new homes. Understanding that path (and its interaction with sync settings and HDFS replication) is what lets a team state, with evidence, what their actual data-loss window is: with default sync-on-ack, effectively zero for acknowledged puts; with deferred sync, a tunable-but-real number someone must own.
The architecture: every piece explained
Top row: the accept path. A client put (usually batched) arrives at the RegionServer hosting the target region. The server acquires region-level resources, stamps the batch with a sequence id, and appends to the WAL: one shared log per server (or per-group with multiwal), entries carrying (region, sequence id, edits), written to an HDFS pipeline and synced according to durability policy — SYNC_WAL (default: ack after pipeline sync) vs ASYNC_WAL (group-synced later; a bounded loss window for throughput). Only after WAL success do edits enter the MemStore — one per store (column family) per region: a ConcurrentSkipListMap of cells sorted by rowkey/family/qualifier/timestamp, sized in heap (or off-heap with recent implementations), with snapshot semantics for flushing (the active segment freezes, a new one accepts writes, the frozen snapshot streams to disk).
Middle row: visibility and persistence. MVCC (the multiversion concurrency control sequencer) gates read visibility: a batch's cells become readable only when its write number completes, so scanners never observe a torn batch; this is also what orders concurrent mutations sensibly without read locks. Flush fires per region when any store hits the flush size (128MB default), per server when global MemStore pressure crosses limits (upper/lower watermarks of heap), on WAL-count pressure (too many unarchived logs → force-flush the regions pinning them), or periodically: the snapshot writes an HFile (sorted blocks, index, bloom filter) and the store's file list updates atomically. WAL rolling cuts a new log at size/time thresholds; a rolled log archives once every region's edits in it have flushed — the retention coupling — and replication tails logs from here too. Compaction feedback closes the LSM loop: flushes mint files, minor compactions fold them, and the store-file count feeds back into flush/blocker logic.
Bottom rows: the guards. Backpressure: 'too many store files' (blocking store files, default ~16) pauses flushes-then-writes for the region until compaction catches up; 'memstore above blocking multiplier' (region MemStore > N× flush size) blocks writes to the region; global watermark breaches force server-wide flushing and throttling. Each blocker is a named metric — the observability contract. Crash recovery: on RegionServer death, the master's SplitWALManager splits its WALs by region (distributed across surviving servers), reassigned regions replay their edits into fresh MemStores (edits below the last-flushed sequence id skip — the flush marker makes replay idempotent), then open for traffic. The ops strip: sync-latency percentiles, flush-queue and blocker counters, WAL-file counts, and hot-region detection as the daily vitals.
End-to-end flow
Trace a write-heavy day on an IoT ingest table (rowkey: device-salt + timestamp; one column family; 400 regions across 20 servers). Steady state: each server absorbs 60k puts/sec in client-batched groups. Per batch: WAL append + pipeline sync (p99 4ms — the dominant latency), MemStore inserts, MVCC completion, ack. MemStores grow to 128MB in ~7 minutes and flush 30MB HFiles (compression); minor compactions fold every ~5 files; WALs roll at 128MB and archive within minutes because flushes keep pace. Blocker metrics: zero. This is the system at its designed rhythm.
14:00 — a datanode's disk degrades; it stays in pipelines but syncs slowly. WAL sync p99 jumps to 90ms; every put on affected servers inherits it. The sync-latency alert names the culprit before clients finish complaining; the datanode is excluded; latency recovers. The lesson institutionalized: WAL sync percentiles are the write path's heartbeat — they page before anything else moves.
16:30 — a firmware bug makes 2% of devices chatty, and their salted keys concentrate on six regions. Those regions flush every 40 seconds, minting small files faster than compaction folds them; store-file counts climb through the blocking threshold; the 'too many store files' blocker pauses writes to those regions in bursts. Clients see region-localized latency spikes; the hot-region dashboard (requests + flush counts per region) points at the six. Tactical response: raise the blocking limit slightly and let a manual major compaction clear the backlog; strategic response: split the hot regions and fix the salting for that device class. Meanwhile the global picture stays healthy — backpressure contained the damage to the hot regions, exactly its job.
02:11 — a RegionServer dies mid-ingest (kernel panic). Its 20 regions held ~800MB of un-flushed MemStore, all durable in WALs. The master distributes WAL splitting; surviving servers each replay a few regions' edits (skipping pre-flush sequence ids), and regions reopen within 90 seconds. The ingest clients, retrying with backoff, lose nothing acknowledged. The morning review reads the recovery like a drill report: split time, replay volume, reassignment latency — numbers the team tracks because the write path's whole durability story is only as real as the last recovery's measurements.