Why architecture matters here
The architecture matters because it is the direct trade between steady-state write throughput and recovery time, and HBase resolves it in favor of throughput. Giving each RegionServer one shared WAL means writes to any region become a single sequential append — the fastest thing a disk or HDFS pipeline can do — with one fsync amortized across all regions' edits. If instead each region kept its own log, a server hosting hundreds of regions would juggle hundreds of open files and sync streams, and write throughput would collapse. So the shared log is the right call for the common case, which is the server running normally.
The bill for that choice comes due exactly when a server dies. Because the log interleaves every region's edits, you cannot simply hand the file to a recovering region — you must read the entire log and demultiplex it, separating each edit by its region so recovered regions can replay only what belongs to them. That demultiplexing is WAL splitting, and its duration is dead time: the affected regions are unavailable until their edits are split out and replayed. Recovery time is therefore a direct function of how much unflushed WAL the dead server held and how fast the split can process it.
Getting this right is what separates a cluster with a few seconds of partial unavailability after a crash from one with minutes of it. The evolution of the mechanism — from the master splitting logs alone, to distributing the work across the whole cluster, to eliminating the demultiplex step entirely with per-region WALs — is a sustained engineering effort to shrink that recovery window. Understanding where the time goes lets an operator size logs, tune split parallelism, and choose the recovery mode that matches their availability target.
It is worth seeing this as a general pattern, not an HBase quirk. Any system that batches many logical streams into one physical log for write efficiency — and many do — pays the same demultiplexing cost at recovery. The design question is always where to spend: HBase spends at recovery so that steady-state writes stay a single fast append, betting correctly that servers run far more often than they crash. The evolution toward WAL-per-region is a hedge on that bet — a way to keep most of the write-side efficiency while cutting the recovery-side cost — and understanding the trade is what lets an operator reason about which mode matches an availability target rather than accepting whatever the defaults happen to give.
The architecture: every piece explained
The pieces begin with failure detection. Every RegionServer holds an ephemeral ZooKeeper node tied to its session; when the server crashes or its session times out, that node vanishes and the HMaster is notified. The master marks the server dead and takes ownership of recovery. The dead server's WAL files live in a known directory on HDFS — durable and readable even though the server that wrote them is gone — and each holds a stream of edits, every entry tagged with its region, row, and a monotonically increasing sequence id.
Splitting is the core work. In the modern distributed log splitting design, the master does not parse the logs itself; it creates split tasks (coordinated through ZooKeeper or, in newer versions, the Procedure framework) and live RegionServers act as split workers, each claiming a log and processing it in parallel. A worker reads its assigned WAL entry by entry, groups the edits by region, and writes each region's edits into a per-region recovered.edits file under that region's directory — sorted by sequence id so replay is ordered. Edits already persisted to an HFile (those whose sequence id is below the region's last flushed id) are skipped, so only genuinely unflushed edits are recovered.
Then reassignment and replay. The master assigns the dead server's regions to live RegionServers. When a RegionServer opens a recovered region, it checks for recovered.edits files, and before it serves a single request it replays those edits into a fresh MemStore, restoring exactly the unflushed state the region had at the moment of the crash, then typically flushes so the recovered.edits can be discarded. Only after replay completes does the region transition to open and start serving. A further evolution, WAL-per-region (or region-grouped WALs), removes the demultiplex step entirely: because a region's edits are already in their own log, recovery becomes a fast rename rather than a full parse-and-regroup, dramatically shrinking the split phase.
The sequence id threaded through every WAL entry is what makes correct recovery possible. Each edit carries a monotonically increasing sequence number, and each region records the highest sequence id it has flushed to an HFile. During splitting, any edit whose sequence id is at or below the region's last-flushed id is already durable and is skipped, so replay applies only the genuinely unflushed tail — never re-applying data that already made it to disk. This is also what makes replay safe to retry: even if a split is re-run after a worker dies mid-task, the sequence-id filter ensures the region converges to exactly the state it had at the crash, no more and no less.
End-to-end flow
Follow a crash. A RegionServer hosting 200 regions dies mid-write; its ZooKeeper session expires and its ephemeral node disappears. The HMaster is notified, marks the server dead, and moves its WAL directory to a splitting location so no one else touches it. The master creates split tasks — one per WAL file — and publishes them for workers to claim.
Live RegionServers pick up the tasks as split workers. Each opens its assigned WAL and streams through it: for every entry it reads the region tag, checks whether the entry's sequence id is above that region's last-flushed id (skipping it if the data is already durable in an HFile), and appends surviving edits to the recovered.edits file in the target region's HDFS directory. Because many workers process many logs at once, the interleaved edits of all 200 regions are demultiplexed in parallel rather than by one machine reading serially. When a worker finishes a log, it marks the task done; the master tracks completion.
As soon as splitting produces a region's recovered.edits, the master reassigns that region to a live RegionServer. That server begins opening the region: it reads the existing HFiles, then finds the recovered.edits file and replays each edit in sequence-id order into a new MemStore, reconstructing the unflushed writes that the crash would otherwise have lost. It then flushes the MemStore to a new HFile and deletes the recovered.edits, and finally marks the region open — now it accepts reads and writes again. From the client's perspective, requests to that region's rows returned errors or blocked from the moment of the crash until this open completes; the total of detection, split, reassign, and replay is the region's outage. When many servers fail together — a rack loss — the master orchestrates all their splits concurrently, and cluster-wide MTTR is bounded by the slowest region's path through this pipeline.
Notice the pipelining in this path. The master does not wait for every log to finish splitting before it begins reassigning regions; as soon as a given region's recovered.edits are complete, that region can be assigned and opened while other logs are still being split. This overlap is deliberate, because it shortens the wall-clock recovery for the earliest-ready regions rather than holding the whole keyspace hostage to the slowest log. It also means recovery is not all-or-nothing: a client whose rows live in an early-recovered region may be served again within seconds, while rows in a region whose edits sat in a large, slow-to-split log wait longer. MTTR is therefore a distribution across regions, not a single number.