HBase RegionServer — WAL, MemStore, Block Cache, and Region Hosting

One process, several tenants of the same heap

Where the RegionServer sits relative to the master, the catalog and HDFS is covered by the HBase overview and the four-component architecture page. Start here instead from the inside: a JVM whose heap is carved into two large, explicitly configured reservations plus whatever is left.

The first reservation is the write side. hbase.regionserver.global.memstore.size (0.4 by default) is the fraction of heap that all write buffers on the server may collectively occupy. The second is the read side: hfile.block.cache.size, also 0.4 by default, is the on-heap cache of decoded store-file blocks. HBase validates the pair at startup and refuses to come up if they sum above 0.8, because the residue is not slack - it is working memory. Response buffers for in-flight RPCs live there, as does per-scanner state, the read and write buffers compaction threads use to stream whole files, the log's append buffers, and the headroom a collector needs to complete a cycle without falling back to a stop-the-world compaction of the old generation.

The write reservation is a shared budget, not a per-region allocation. Buffers are accounted per store, meaning per region per column family: a server holding 120 regions of a two-family table is tracking 240 independent flush units against one pool. That arithmetic is worth doing explicitly, because it usually inverts the mental model people arrive with. On a 32 GB heap the budget is about 12.8 GB; 240 stores each allowed to reach the 128 MB per-store threshold would want 30 GB. The per-store threshold is therefore not what normally fires. On any busy server with a realistic region count, the global watermark fires first and the per-store size is a ceiling that only the handful of genuinely hot stores ever reach.

The split is zero-sum, and you bet before you know the workload

Every point of heap given to write buffering is a point taken from caching, and the two knobs move in opposite directions for opposite workloads. A write-dominated table wants large buffers: fewer, larger flushed files, which means less work handed downstream to compaction. A read-dominated table wants the cache, because the difference between a cached block and an HDFS round trip is three orders of magnitude and shows up directly in the tail. A cluster that serves both from the same servers is being asked to choose, and whichever way it chooses, one of its two workloads is running degraded.

The write side has two watermarks rather than one, and the gap between them is where the server does its polite work. Below hbase.regionserver.global.memstore.size.lower.limit nothing special happens. Between the lower limit and the global size, the server starts selecting the largest buffers and flushing them in the background - writes still complete, latency rises a little, and the only visible sign is a non-empty flush queue. Above the upper limit the server stops accepting mutations entirely until it gets back underneath. That state has its own counter, and the distinction between "flushing hard" and "blocking updates" is the single most useful discrimination available when a write workload goes intermittent.

Moving memory off the heap changes the arithmetic rather than removing the tension. With an off-heap cache, only the small L1 index remains on-heap and the block-cache fraction can be cut back sharply, which frees heap for buffering without shrinking the effective cache at all - see BucketCache for how that tier is organised, and the block cache page for the priority tiers and eviction policy that decide what actually stays resident.

<!-- read-leaning server, on-heap cache -->
<property><name>hbase.regionserver.global.memstore.size</name><value>0.25</value></property>
<property><name>hfile.block.cache.size</name><value>0.55</value></property>

<!-- ingest-leaning server -->
<property><name>hbase.regionserver.global.memstore.size</name><value>0.55</value></property>
<property><name>hfile.block.cache.size</name><value>0.25</value></property>

<!-- the pair is validated at startup; their sum must stay at or below 0.8 -->

A mutation occupies a handler for the whole sync

The mechanics of the write path - log append, buffer insert, sequence ids, backpressure - belong to the write path article, and the durability settings that decide whether the sync is on the critical path at all belong to WAL durability levels. What matters at process altitude is a scheduling fact those pages do not need to dwell on: the thread that accepted the request is parked for the entire duration of the log sync, and it is one of a small, fixed number of such threads.

Follow one: a handler picks the request off a queue, resolves the target region, acquires row locks in a deterministic order so concurrent batches cannot deadlock, hands its edits to the log's append machinery, and then waits. A separate group of threads does the actual append and sync against HDFS, batching across every handler that is waiting at that moment - group commit, which is why per-write sync cost falls as concurrency rises. When the sync returns, the handler inserts into the buffer, completes its multiversion write number so the edits become visible to new scanners, acknowledges, and only then goes back to the pool.

The consequence is that sync latency does not merely add to write latency; it converts directly into handler occupancy, and handlers are the scarcer resource. With the default pool of thirty and a healthy 4 ms sync, a server can hold roughly 7,500 syncing writes per second in flight before the pool itself becomes the ceiling. Let one degraded DataNode push the sync to 90 ms and the same pool tops out near 330. No configuration changed, no region moved, and the server's own CPU is idle - it is simply that every handler is asleep waiting on a disk somewhere else in the rack.

Why there are several flush triggers instead of one

A flush looks like a single event but is reachable from at least four directions, and they exist because each one is protecting a different scarce resource. Knowing which one fired is most of the diagnosis.

Per-store size (hbase.hregion.memstore.flush.size, 128 MB) bounds the size of an individual flush and therefore the size of the file it produces. It protects the shape of the data on disk. The global watermark protects the heap, and as shown above it is the trigger that actually dominates on a well-populated server. Log count (hbase.regionserver.maxlogs) protects recovery time: a log file can only be archived once every edit inside it has been flushed somewhere, so one barely-written region can pin a long tail of logs that are otherwise entirely obsolete. When the count crosses the limit the server force-flushes whichever regions are holding the oldest logs, which is why an idle table can produce a burst of tiny files for no apparent reason. The periodic trigger (hbase.regionserver.optionalcacheflushinterval, an hour by default) bounds staleness, so edits do not sit in memory indefinitely on a region nobody writes to. On top of those, region close, split, snapshot capture and graceful shutdown all flush explicitly.

A fifth guard is per-region rather than per-server and is often mistaken for the global one. hbase.hregion.memstore.block.multiplier blocks writes to a single region whose buffer has grown past a multiple of the per-store threshold - a local valve that stops one runaway region from consuming the whole server budget before the global watermark notices. Writes stalling on one region while the rest of the server is comfortable is that valve, or the store-file blocker described under compaction; writes stalling everywhere at once is the global watermark. They present identically to a client and have nothing to do with each other.

A read is an assembly job, and the assembly holds resources

Serving a Get is not a lookup, it is a merge. The server builds a scanner per region, under it a scanner per store, and under that one source scanner for each place a newer version could be hiding: the active write buffer, any snapshot segment currently being flushed, and every store file the region owns. Those sources feed a heap ordered by cell key, and the merge walks it until it has the newest visible version of each requested cell. Which files can be skipped is decided by bloom filters and the file's internal index - see bloom filters and the HFile format - and whether the surviving blocks cost a disk seek is decided by the cache.

Two properties of that assembly are process concerns rather than read-path concerns. The first is the read point: a scanner fixes a multiversion read number when it opens, and anything committed afterwards is invisible to it. That is what makes each row a coherent snapshot of itself. The second is that an open scanner is a resource holder. It pins the store files it opened, so a file that compaction has already superseded cannot be deleted until the last scanner over it closes; it holds references to cached blocks; and it holds a lease. If a client stops calling for results and the lease (hbase.client.scanner.timeout.period) expires, the server discards the scanner and the next call fails with an unknown-scanner error - a client-side pause showing up as a server-side error, which is why slow client-side processing inside a scan loop is a genuinely common production surprise.

The RPC layer, and how one scan starves every point read

All of the above runs on a fixed thread pool sized by hbase.regionserver.handler.count, thirty by default. Requests arrive, land in call queues, and wait for a handler. Everything about the server's behaviour under contention follows from the fact that this pool is small and that occupancy time varies by three orders of magnitude across request types.

A Get that hits cache occupies a handler for a fraction of a millisecond. A scan RPC returning a large batch can occupy one for hundreds. With a single undifferentiated pool, thirty concurrent batch scans take every handler, and a 2 ms Get sits in the queue behind them. The Get's own service time has not changed at all - what exploded is its queue time, and this is precisely why latency percentiles alone will mislead you here. Queue-time metrics and call-queue length name the problem; process-time metrics say everything is fine.

HBase's structural answer is to stop sharing. hbase.ipc.server.callqueue.read.ratio partitions the handlers into separate read and write pools so a write stall cannot consume the readers, and hbase.ipc.server.callqueue.scan.ratio carves scans out of the read share again, so a scan flood can exhaust its own slice while point reads keep theirs. hbase.ipc.server.callqueue.handler.factor controls how many queues back those handlers - a value of one gives each handler its own queue and removes contention on the queue itself, at the cost of losing work-stealing between them. Queues are also bounded: past a length limit or the aggregate byte limit (hbase.ipc.server.max.callqueue.size, a gigabyte by default) the server rejects with a call-queue-too-big error. That is deliberate load shedding, it is retriable, and confusing it with a timeout sends an investigation in exactly the wrong direction.

Beyond partitioning, the levers are to make individual RPCs cheaper - lower client-side scan caching returns smaller batches and releases the handler sooner - or to move the offending workload off the server entirely, via RegionServer groups for physical separation or quota throttling to meter it in place.

Opening a region is work, and it is charged at the worst moment

A region is owned by exactly one server at a time, and moving that ownership is not free. To open one, the server reads the region's metadata, opens each store, reads the trailer of every store file and loads that file's index and bloom blocks, replays any recovered edits left behind by a previous failure, updates the catalog, and only then transitions the region to open and starts answering for those keys. The cost scales with store-file count, not data size: three hundred regions averaging five files each means fifteen hundred files to open before the server is fully useful.

The invisible part of the bill is cache. A region arriving on a new host arrives cold - its blocks are not in that server's cache, and its files are still physically located on the DataNodes near its old host, so reads cross the network until a compaction happens to rewrite them locally. This is why a rolling restart depresses read latency for far longer than the restart itself takes, and why aggressive rebalancing can cost more than the imbalance it corrects. Placement decisions and their cost model live in the balancer and the cost-function implementation; how many regions a server should carry is bounded by all of the above plus the buffer arithmetic from the first section, which is what keeps the practical ceiling in the low hundreds rather than the thousands. The orchestration that drives assignment is the master's, via durable procedures and the catalog.

The session is the liveness signal, and a pause is indistinguishable from death

A RegionServer proves it is alive by holding an ephemeral coordination node in ZooKeeper. It is not a health check in any meaningful sense - nothing inspects whether the server is serving well, only whether its session is being renewed. When the session expires, the node vanishes, the master concludes the server is gone, and recovery starts: logs are split and regions are reassigned elsewhere (the full path is WAL splitting).

The timeout is zookeeper.session.timeout, and HBase requests a comparatively generous value because it knows its own processes pause. There is a trap here that catches people: the ensemble negotiates the effective timeout and clamps it to its own bounds, which by default top out at twenty times the ensemble's tick interval. With a 2-second tick, a requested 90 seconds is silently reduced to 40 unless the ensemble's maximum is raised to match. A server can therefore be operating with a detection window a fraction of what its own configuration file claims.

The Juliet pause

Now the failure this whole design cannot avoid. A full garbage collection that runs longer than the timeout stops the heartbeat exactly as thoroughly as a kernel panic does. The session expires, the master buries the server, splits its logs and hands its regions to peers. Then the collection finishes and the process resumes, believing it still owns regions that now have a different owner - alive, but pronounced dead and already buried, which is where the name comes from. HBase resolves it in the only safe direction: on discovering its session is gone, the server aborts itself, and the master rejects its reports outright. Correctness survives; the server does not. A healthy machine has been destroyed by a pause.

The knob is genuinely two-sided and there is no setting that is right. Raise the timeout and you tolerate long pauses but add that entire interval to the recovery time of every real crash, during which the affected key ranges answer nothing. Lower it and real crashes recover fast but a thirty-second collection kills a working server and triggers a recovery that was never needed. The timeout only chooses which failure you prefer; the actual fix is upstream, in making the pauses shorter.

Why RegionServer heaps stayed small for so long

The conventional sizing - a heap in the tens of gigabytes even on machines with far more RAM - looks like timidity until you connect it to the previous section. Pause duration scaled with heap size; the detection window did not. Past a certain point, enlarging the heap to hold more cache made the server more likely to be killed for a pause than to benefit from the cache, and the failure was not gradual.

Two properties of the workload made it worse than a generic large-heap Java service. Write buffers churn in a fragmentation-shaped pattern: cells of wildly varying sizes allocated continuously and then freed in bulk at flush, leaving a heap full of holes. That is what MSLAB (hbase.hregion.memstore.mslab.enabled) exists to fix, allocating buffered cells out of fixed-size chunks so a flush releases whole chunks rather than scattered fragments - a fragmentation remedy specifically, not a throughput one. And an on-heap block cache is long-lived by design and large by intent, which is precisely the population a generational collector must repeatedly trace and copy.

The escape was to move the memory rather than grow the heap. The cache went off-heap first, and write buffers followed, so a modern server can run a modest heap alongside a large off-heap footprint that the collector never walks. Newer low-pause collectors shift the arithmetic again, but they relocate the constraint rather than removing it - the detection window is still a fixed number and the pause is still the thing that has to fit inside it. Collector selection and pause targets are their own topic.

Reading a sick RegionServer

The server exports far more metrics than anyone can watch. The useful ones are useful in pairs, because almost every diagnosis here is a discrimination between two causes that produce the same client-visible symptom. Thresholds, alert routing and on-call plumbing are a separate concern, covered in HBase alerting practice; what follows is the differential.

Two habits make the table usable. Keep the per-region breakdown, not just the server aggregate, because the most common shape of an HBase problem is one region misbehaving inside an otherwise healthy server and the aggregate hides it entirely - hotspotting is the usual reason. And record the server's own log alongside the metrics: forced flushes, blocked updates, aborted scanners and session expiry all announce themselves in plain text with the region name attached, which is information no counter carries.

The failure modes worth rehearsing

The handler famine. A batch job opens wide scans against a table that shares servers with an interactive one. Handlers fill, queue time climbs, and every tenant on those servers degrades together. It is not a capacity problem and adding servers barely helps; the fix is partitioning - separate handler pools, separate groups, or a quota.

The slow disk two hops away. One DataNode degrades without failing. Log syncs that touch it slow down, handlers stay parked longer, and a server whose CPU is idle stops accepting writes. The RegionServer looks broken and is not.

The pause spiral. Heap pressure lengthens collections, a collection overruns the session, the server is declared dead, its regions land on peers that were already loaded, and their heaps come under pressure. This is the failure mode that turns one sick server into a cluster event, and it is why pause duration is worth watching even while everything is nominally healthy.

The cold restart. A rolling restart moves every region twice. Each new host opens files, starts with an empty cache and reads non-locally until compaction restores locality. Read latency stays elevated for hours after the operation is "complete", which is a planning fact rather than a fault.

Too many regions. Region count is a memory commitment before it is anything else - buffers per store, file handles, index and bloom blocks pinned per open file, and a heartbeat report that grows with it. A server carrying too many regions flushes constantly at small sizes, hands compaction an unreasonable file population, and opens slowly after every restart. The normalizer addresses the sizing side of this.