HBase is an LSM-tree, and the MemStore is its mutable layer. Every write lands in memory, sorted, and stays there until something decides to turn it into an immutable file on HDFS. That decision -- when to flush, which regions to flush, and what to do when flushes cannot keep up -- drives most of the performance behaviour operators actually experience: file counts that overwhelm compaction, heap dominated by buffered writes at the expense of the block cache, and the write stalls that turn a busy cluster into an unresponsive one. This article covers the MemStore's structure, every trigger that causes a flush, what the flush actually does, and how the stall cascade develops. Durability of the write-ahead log and the merging of HFiles after they land are covered by this category's WAL and compaction articles.
Where the MemStore sits
A write arrives at a RegionServer and does two things before it is acknowledged: it is appended to the write-ahead log, and it is inserted into the MemStore for the target store. Nothing is written to HDFS as a data file at this point. The client's acknowledgement means 'durably logged and visible in memory', not 'persisted as an HFile'.
Reads then have to merge. A scanner for a region consults the MemStore, the block cache, and every HFile in the store that might contain relevant cells, combining them in cell order so the newest version of each cell wins. This is why a region with a large MemStore and many small HFiles is slow to read as well as expensive to maintain -- the merge has more inputs.
The granularity is per store, which means per column family per region. A region with three column families has three MemStores. A RegionServer hosting two hundred regions with two families each is managing four hundred of them out of one heap, which is the arithmetic behind most memory-pressure problems in HBase and behind the standing advice to keep column families few.
The data structure, and why it is a skip list
A MemStore holds cells in a concurrent skip list, ordered by the same comparator that orders cells on disk: row key, then column family, then qualifier, then timestamp descending, then type. Keeping the in-memory structure in the same order as the on-disk format is what makes a flush a sequential write with no sort step, and what lets a scanner merge memory and files without buffering.
A skip list rather than a balanced tree because it supports concurrent readers and writers without global locking, which matters when many handler threads insert into the same store while scanners iterate it. The cost is pointer-chasing and per-entry overhead -- a skip-list node carries several references per cell, so the memory a MemStore occupies is meaningfully larger than the size of the data in it.
That overhead led to MSLAB, the MemStore-Local Allocation Buffer. Without it, cells of wildly varying sizes are allocated individually on the heap, and when a flush frees them the result is a fragmented old generation and, eventually, promotion failures and long garbage-collection pauses. MSLAB instead copies cell data into fixed-size chunks -- two megabytes by default -- so allocation and release happen at chunk granularity and fragmentation largely disappears. A chunk pool recycles the chunks so steady-state allocation approaches zero, and off-heap MSLAB moves the chunks out of the Java heap entirely, which is the standard configuration for write-heavy clusters with large heaps.
Every trigger that causes a flush
Operators usually know the first one and are surprised by the rest. All of these fire in production.
Per-store size. When a single MemStore exceeds hbase.hregion.memstore.flush.size -- 128 MB by default -- the region is flushed. This is the intended, healthy trigger and produces HFiles of a predictable size.
Region block multiplier. If a MemStore reaches hbase.hregion.memstore.block.multiplier times the flush size -- four times 128 MB, so 512 MB by default -- writes to that region are blocked until a flush completes. This is a safety valve against a single hot region consuming the heap, and hitting it is a symptom, not a normal event.
Global heap pressure. The sum of all MemStores on the RegionServer is bounded by hbase.regionserver.global.memstore.size, 40 percent of heap by default. Crossing a lower threshold triggers forced flushes of the largest MemStores, in size order, until the total drops. Crossing the upper limit blocks writes across the entire RegionServer, not just one region. Flushes caused this way are often small, which is how a cluster ends up producing many undersized HFiles under load.
WAL count. A WAL file cannot be discarded while it contains edits not yet persisted in an HFile. When the number of live WAL files exceeds the configured maximum, HBase forces flushes of whichever regions are pinning the oldest WALs -- regardless of how small their MemStores are. On clusters with many regions receiving trickles of writes, this is frequently the dominant flush trigger and the reason for a steady stream of tiny HFiles.
Periodic flush. A background thread flushes any MemStore older than hbase.regionserver.optionalcacheflushinterval, one hour by default, with jitter so that regions do not all flush together. This bounds recovery time by ensuring old edits do not sit in memory indefinitely.
Explicit and structural. An administrative flush, a region close, a split, a merge, or a snapshot all flush first.
What a flush actually does
The flush is designed so that writes do not stop while it runs.
First the active MemStore is snapshotted: the current structure is set aside as an immutable snapshot and a fresh empty one takes its place. Incoming writes go to the new one immediately, so the write path pauses only for the brief moment of the swap. Reads consult both until the flush completes.
Then the snapshot is written to a new HFile in a temporary location, in the cell order it already has, with Bloom filters and block index built as it goes. When the file is complete it is moved into the store's directory and added to the store's file list atomically, the snapshot is discarded, and the memory is released. Finally the sequence-id watermark advances, which is what tells the WAL machinery that everything up to that point is now durable in HFiles and the corresponding WAL files can be archived.
If the RegionServer dies mid-flush, nothing is lost and nothing is half-applied: the temporary file is orphaned and cleaned up, the sequence id never advanced, and the edits are replayed from the WAL during recovery. This is the property that lets HBase acknowledge writes before they are in an HFile.
One consequence worth internalising: a flush creates exactly one HFile per store. Flush frequency therefore sets the file-creation rate, and the file creation rate is what compaction has to keep up with. Every tuning decision in this article is ultimately about that ratio.
Multiple column families flush together
The unit of flushing is historically the region, not the store. When a region is flushed, every column family's MemStore in it is flushed -- including ones holding a few kilobytes.
That is why unbalanced column families are the classic HBase schema mistake. Suppose one family receives heavy writes and a second receives occasional metadata updates. The heavy family reaches the flush threshold constantly, and each of those flushes also writes a tiny HFile for the metadata family. The small-file problem is created by a family that is barely used, and compaction then has to deal with it forever.
Later versions soften this with a per-family flush policy that flushes only the stores above a size lower bound, leaving small ones in memory. It helps, and it does not remove the underlying coupling -- the WAL-count trigger, for instance, still forces the whole region. The durable advice stands: use one column family unless you have a specific reason, and never let two families have very different write rates. Two or three families with comparable traffic is fine; one hot and one cold is a structural problem no configuration fully fixes.
In-memory compaction
HBase 2 introduced a compacting MemStore, which changes what happens between flushes. Instead of one growing skip list, the MemStore becomes an active segment plus a pipeline of immutable segments. When the active segment fills, it is pushed into the pipeline and a new active segment starts -- and segments in the pipeline can be merged, or compacted, in memory.
The policies are worth knowing by name because the choice is workload-dependent. None restores the classic single-skip-list behaviour. Basic merges segments and flattens their indexes without removing redundant cells, which reduces overhead cheaply. Eager additionally eliminates cells that are already superseded -- overwritten values, deletes, versions beyond the family's limit -- so what eventually reaches disk is much smaller. Adaptive measures the redundancy it observes and chooses between them.
The win is largest for workloads with heavy churn: counters, status fields updated repeatedly, anything where the same cell is written many times before it is ever read. There, eager in-memory compaction can drop most of the data before it is ever written, which reduces flush frequency, HFile count, compaction work and write amplification together. For append-only workloads with unique keys there is nothing to eliminate, and eager compaction only spends CPU -- basic or none is the right setting.
The pipeline also enabled a more compact in-memory index, which stores cell metadata in chunks rather than as individual objects. That reduces per-cell overhead substantially and works with off-heap chunks, which is what makes very large MemStores practical without punishing garbage collection.
The write-stall cascade
This is the failure mode operators need to recognise on sight, because every stage looks like a different problem.
It starts with compaction falling behind. Each flush adds an HFile; if compaction cannot merge them as fast as flushes create them, the file count in a store climbs. When it reaches hbase.hstore.blockingStoreFiles -- sixteen in recent versions, lower in older ones -- HBase stops flushing that store, on the reasoning that adding more files would make the situation worse.
Now the MemStore cannot drain. It grows to the block multiplier threshold, and writes to that region block. If the region is hot, its MemStore is a large share of the server's total, so global MemStore pressure rises and other regions are force-flushed to compensate -- creating more files, in other stores, accelerating the same problem elsewhere. Eventually the server-wide MemStore limit is hit and every write to the RegionServer blocks. Clients see timeouts and region-too-busy responses, retries pile on, and the server may be marked dead and have its regions reassigned to neighbours that are already struggling.
The signature in the logs is unmistakable once you know it: messages about too many store files delaying a flush, followed by messages about blocking updates for a region, followed by global memstore pressure. The metric equivalents are a rising store file count, a growing flush queue, and non-zero blocked-request and updates-blocked-time counters.
The fixes are ordered by where the bottleneck actually is: give compaction more threads and more I/O headroom; reduce the flush rate by raising the flush size or reducing region count per server so each region has room; enable in-memory compaction if the workload has churn; fix a hot key distribution that is concentrating writes; and only as a stopgap raise the blocking file threshold, which buys time by allowing reads to degrade instead of writes.
Sizing MemStore against the block cache
A RegionServer's heap is divided between MemStores and the block cache, and the two budgets are configured independently with a hard constraint that their sum stays below roughly eighty percent of the heap -- HBase refuses to start otherwise, which is a usefully blunt guard.
The default split of forty percent each suits a mixed workload. A write-dominated cluster can shift towards MemStore, reducing flush frequency and file creation. A read-dominated cluster shifts towards the block cache, and should also consider a bucket cache off-heap so that a large cache does not create a large heap to collect.
The second constraint is regions per server times flush size. With the default 128 MB flush size, a server hosting 300 actively-written regions would need 38 GB of MemStore to let every region reach its threshold. If the global budget is smaller -- and it always is -- then most flushes will be triggered by global pressure rather than by size, and they will be small. This is the arithmetic behind the standard guidance to keep the number of regions per RegionServer modest, in the low hundreds, and to prefer fewer larger regions over many small ones for write-heavy tables.
The practical method: count actively-written regions per server, multiply by the flush size, compare against the global MemStore budget, and if the product exceeds the budget by a wide margin, expect global-pressure flushes and either reduce the region count, raise the heap, or accept and plan for smaller HFiles with more compaction.
Diagnostics — what to look at, in order
Store file count per store. The leading indicator for everything in the stall cascade. A steady climb means compaction is losing.
Flush queue length and flush time. A queue that is never empty means flushes are being requested faster than they complete; long flush times point at HDFS write latency rather than at HBase.
MemStore size, global and per region. Compare against the configured limits. Sitting near the lower global threshold means most flushes are pressure-driven rather than size-driven.
Blocked request count and updates-blocked time. Any non-zero value is writes being stopped. These should be alerted on, not merely graphed.
Average HFile size at creation. Not always exposed directly, and worth deriving. If flushes are producing files far below the flush size, the trigger is not size -- it is WAL count or global pressure, and the remedy is different.
The logs. HBase states its reasons plainly: which region it flushed and why, when it delayed a flush for too many store files, when it blocked updates. Grepping for the flush and block messages during an incident is faster than any dashboard, and the reason string is the diagnosis.
Practical guidance
One column family unless proven otherwise, and never families with very different write rates.
Keep regions per server in the low hundreds for write-heavy tables, and size regions so that the MemStore budget divided by active regions is a meaningful fraction of the flush size.
Enable off-heap MSLAB with a chunk pool on large-heap, write-heavy servers; it is the difference between predictable and unpredictable pause times.
Match in-memory compaction to churn: eager or adaptive where cells are overwritten repeatedly, basic or none for append-only unique-key workloads.
Watch the file count, not the MemStore. The MemStore is where the symptom appears; store file count is where the cause is.
Fix hot keys at the schema level. No flush tuning saves a table whose row key sends every write to one region -- salting or a better-distributed leading key is the actual fix, and it belongs in the schema conversation rather than in the tuning one.