Why it matters
Write latency in HBase is dominated by WAL sync. Understanding the WAL is understanding where write latency comes from and how to reduce it without sacrificing durability. This is one of those topics where getting it right is easy but getting it wrong loses data — worth investing time in.
WAL performance also affects the whole cluster's write throughput. A slow WAL on one RegionServer caps its write rate; a slow HDFS underneath caps every RegionServer's WAL.
The architecture
Physically, a WAL is an HDFS file. Each RegionServer typically has one WAL file at a time. Every mutation from every region hosted on that RegionServer is appended to this file. When the file reaches a size threshold, it is rolled: the current file is closed and a new one is opened.
Records in the WAL are keyed by region and by sequence number, so replay can distinguish mutations for different regions and apply them in order.
How it works end to end
On write, the RegionServer appends the mutation to the WAL, waits for the sync policy's completion signal, and then applies the mutation to the memstore. The client is not acknowledged until the WAL sync completes. The sync policy determines how strict this durability guarantee is.
Sync policies: SYNC_WAL (default) syncs to HDFS on every write, using hflush semantics that push bytes to all replicas but do not fsync to disk. FSYNC_WAL forces fsync on every write, guaranteeing disk durability but at higher latency. ASYNC_WAL batches syncs periodically, giving best latency but potentially losing recent writes on crash.
WAL replay on recovery works like this: when a RegionServer dies, the HMaster reassigns its regions to other RegionServers. Each recovering RegionServer reads the crashed WAL, extracts records for its assigned regions, and replays them into its memstore. Once replay finishes, the recovering RegionServer can serve reads and writes.