HBase Regions and Splits — How Data Partitioning Grows and Rebalances

Why it matters

Region splits are the primary mechanism by which HBase scales. Without them, a growing table would grow into a single region and become a single-server bottleneck. With them, tables scale linearly as data grows: more data means more regions, and more regions spread across more RegionServers.

The cost of leaving splits entirely automatic is that hot tables can generate constant splits, which affect availability. Understanding split policies lets you choose behavior that matches your workload's tolerance for brief region unavailability.

The architecture

Each region has a size threshold (default 10 GB) called the split threshold. When any store file (HFile plus memstore) within the region exceeds this size, the RegionServer schedules a split. The split finds a midpoint row key that divides the region roughly in half.

Actual splitting is fast because it does not rewrite data. The two daughter regions share HFiles with the parent, using reference files that point into the parent's HFiles with byte-range offsets. The parent is marked offline and the daughters take over. Later, compaction rewrites the reference files into standalone HFiles.

How it works end to end

A split has three separable questions, and conflating them is the source of most confusion. When does the trigger fire - that is the split policy, evaluated by the RegionServer after every memstore flush and every compaction. Where does the region divide - that is split point selection, which reads the block index of the region's largest store. What actually moves - almost nothing, because a split writes a handful of tiny pointer files and defers every byte of real data movement to a later compaction.

Manual splits via the HBase shell can force a specific split point, which is useful when a particular row key range has become hot and the automatic policy would leave it as one large region. Starting a table with several regions instead of one - pre-splitting - is the other half of the story, and it belongs to key design rather than to split mechanics; see HBase hotspotting for split points derived from salted, hashed and reordered keys.

Choosing the split point

The RegionServer does not compute a median row key, and it does not scan the region to find one. It asks each store (one per column family) for its size, picks the largest, and asks that store's largest HFile for the midkey of its multi-level block index. The block index already records the first row key of every data block, so the midkey is the first key of the block sitting halfway down the index - an O(1) lookup against a structure that is already in memory. That is why split point selection costs nothing even on a 10 GB region.

Two consequences follow from using a real key rather than a computed midpoint. First, the split point is always an existing row boundary. HBase never divides a row across two regions, so a single row that has grown to gigabytes across many columns is unsplittable - the midkey resolves to the region's only row, the split is rejected, and the region grows without bound. Second, the midkey measures bytes written, not writes arriving, and on a monotonically increasing key those two diverge completely.

Work the monotonic case through. A region covering timestamps from midnight to noon reaches the threshold and splits at its byte midpoint, roughly 06:00. Every subsequent write carries a timestamp later than noon, so it lands in the upper daughter; the lower daughter is frozen the instant it is created and will never take another write. The upper daughter refills at the same rate the parent did and splits again, producing another dead half. The split machinery is working exactly as designed and still converts one hot region into a chain of cold regions plus one hot region. No split policy fixes this - the fix is in the row key, and it is covered in HBase hotspotting.

A split is a metadata operation, not a data rewrite

When the trigger fires, the RegionServer creates a staging directory under the region directory (classically .splits) containing a directory for each daughter. For every store file in the parent it writes two Reference files, one into each daughter. A Reference file is not a copy. Its entire payload is the split key and a one-byte flag saying whether this reference covers the top half or the bottom half of the parent file, and its name encodes the parent file and the parent's encoded region name. A few dozen bytes, per store file, per daughter.

That is the whole reason a 10 GB region splits in well under a second while a 100 MB region takes the same time. The parent is closed, the daughter directories are moved into place, hbase:meta is updated, and the daughters are opened - typically a sub-second window during which that key range is unavailable and clients see NotServingRegionException. What happens to the meta rows and how clients recover their cached locations is the subject of the hbase:meta table; what follows here is the storage side.

A daughter serving reads through a Reference opens the parent's HFile. A bottom reference seeks to the file start and stops at the split key; a top reference seeks to the split key and reads to the end. Both daughters hold handles on the same physical file, and both pay for the parent's full-size block index and bloom filter to reach half as much data. During this window read amplification is real and measurable: a bloom filter sized for the parent's key count now answers questions about half that many keys, so its false positive rate against a daughter's working set is worse than the configured target. See HBase bloom filters for how that sizing works.

The window closes at the next compaction. Compaction on a daughter reads through its References and writes a standalone HFile containing only that daughter's rows, after which nothing points at the parent's files. This is the real cost of the split, paid asynchronously by the compaction queue - see HBase compaction. A cluster that splits faster than it compacts accumulates References, and a region carrying References cannot be split again, so the split machinery stalls behind the compaction backlog.

Split policies: when the trigger fires

The policy class is checked after every flush and compaction and answers a single boolean. It is set cluster-wide with hbase.regionserver.region.split.policy or per table in the table descriptor.

The constant-size family

ConstantSizeRegionSplitPolicy splits when the largest store exceeds hbase.hregion.max.filesize, default 10 GB. Predictable and easy to reason about, with one bad property: a brand-new table is a single region on a single server, and it stays that way until it has absorbed 10 GB. For the first several hours of a table's life the cluster has one server doing all the work.

The increasing-to-upper-bound family

IncreasingToUpperBoundRegionSplitPolicy exists to fix exactly that. Its threshold is min(hbase.hregion.max.filesize, initialSize * R^3), where R is the number of regions of this table on this RegionServer and initialSize defaults to twice hbase.hregion.memstore.flush.size. With the default 128 MB flush size that is 256 MB, and the curve is steep:

A young table therefore splits aggressively - the first split at 256 MB, the second at 2 GB - and by the fourth region per server the cube has overshot the ceiling and the policy is indistinguishable from constant-size. That is the intended shape: reach a handful of regions per server quickly, then stop churning. SteppingSplitPolicy, the default in HBase 2.x, produces the same shape without the curve: if the table has exactly one region on this server the threshold is initialSize, otherwise it is hbase.hregion.max.filesize.

The special-purpose policies

KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy take the midkey the block index returned and truncate it to a fixed byte length or to a delimiter, guaranteeing that every row sharing a prefix stays in one region. That matters when a coprocessor or a multi-row atomic operation assumes co-location, since HBase only offers atomicity within a region. BusyRegionSplitPolicy ignores size entirely and triggers on the fraction of time a region spent blocked on requests, which is the only automatic policy that reacts to a small-but-scorching region. DisabledRegionSplitPolicy never splits.

Configuration and shell operations

<!-- cluster defaults -->
<property><name>hbase.hregion.max.filesize</name><value>10737418240</value></property>
<property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
<property>
  <name>hbase.regionserver.region.split.policy</name>
  <value>org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy</value>
</property>
<!-- a RegionServer stops splitting once it hosts this many regions -->
<property><name>hbase.regionserver.regionSplitLimit</name><value>1000</value></property>

# per-table policy, overriding the cluster default
alter 'events', CONFIGURATION => {
  'hbase.hregion.max.filesize' => '21474836480',
  'hbase.regionserver.region.split.policy' =>
    'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' }

split 'events'                       # every region, at its own midkey
split 'events,\x04user#881,169...'   # one region, at its midkey
split 'events', '\x07'               # the table, at an explicit row key

merge_region 'a1b2c3d4', 'e5f6a7b8'  # two ADJACENT encoded region names

Splitting an already-split region fails while it still holds Reference files, so a script that walks a table issuing splits must tolerate and retry that error rather than treating it as fatal. The same is true immediately after a bulk load, where the loaded files have not yet been compacted into the target regions - see HBase bulk load.

Turning automatic splits off on purpose

Teams running latency-sensitive workloads on a well-understood key space frequently disable automatic splitting and manage it themselves. The reason is timing. An automatic split fires whenever the data happens to cross the threshold, which is disproportionately likely to be during peak write load - and it briefly takes the region offline, invalidates every client's cached location for it, and queues compaction work on two daughters at precisely the moment the compaction queue is already deepest. A p99 that is otherwise flat develops spikes that correlate with nothing in the application.

The managed-split pattern is: create the table pre-split at chosen boundaries, set DisabledRegionSplitPolicy, raise hbase.hregion.max.filesize well above the intended region size as a backstop, and run a scheduled job that inspects region sizes and issues explicit split commands during a maintenance window. You get splits at a time you chose, at split points you chose, with compaction fallout landing when the cluster is idle.

The cost is that you now own the growth curve. Forget the scheduled job for a quarter and a region reaches 100 GB, at which point a major compaction on it reads and rewrites 100 GB, its store files are almost certainly no longer local to the server serving them, and the split you eventually issue produces two 50 GB daughters whose References take hours to resolve. Disabling splits converts an availability problem you cannot schedule into an operational obligation you must not forget.

What splitting costs you operationally

Region count only ever ratchets up

Every split adds a region, and nothing removes one automatically. Each region on a RegionServer carries fixed cost regardless of how much data it holds: an open memstore per column family, open file handles per store file, a row in hbase:meta, and a slot in the balancer's cost computation. The memstore cost is the one that bites first. MSLAB allocates memstore in 2 MB chunks per store, so a three-family region reserves roughly 6 MB of heap before it holds a single cell.

Worse, hbase.regionserver.global.memstore.size caps total memstore at 40% of heap. On a 32 GB heap that is about 13 GB; spread across 1000 regions each memstore gets around 13 MB, so the global ceiling forces flushes long before the per-region hbase.hregion.memstore.flush.size of 128 MB is reached. The result is many small HFiles, which means constant compaction, which means the compaction queue never drains. Practical guidance lands around 100-200 regions per server. Sizing that deliberately is region count and sizing; reclaiming regions that deletes have hollowed out is the region normalizer.

Regions in transition

Between the parent closing and both daughters opening, three regions are in transition. In healthy operation the Master UI's RIT count returns to zero within seconds. A RegionServer that dies mid-split leaves regions stuck in SPLITTING or OPENING, and a region stuck there for minutes is an incident, not a delay - the key range is unavailable the entire time. Recovery is HBCK2 territory; diagnosis is covered in regions in transition troubleshooting.

Locality survives the split and dies later

A fresh split keeps full data locality for free: the References point at HFiles that are already on the local DataNode because the splitting RegionServer wrote them. Locality is lost afterwards, when the balancer relocates a daughter to even out region counts. The new host reads References - and later, compacted HFiles - across the network until a major compaction rewrites the blocks with the new host as the local replica. This is why a burst of splits followed by a balancer run degrades read latency for hours. The balancer's locality cost function is covered in the HBase balancer and region data locality.

Merge: the inverse operation

Merge exists because splits are one-way. A table that was heavily pre-split, or one whose old rows have been removed by deletes and TTL expiry, ends up with many regions holding almost nothing while still paying the full per-region overhead above. Merging fuses two regions into one and reclaims it.

The one mechanical constraint worth knowing here is that only adjacent regions can merge, because a region owns a contiguous row key range and the result must also be contiguous. The shell's merge_region takes two encoded region names and refuses non-neighbours. Deciding which regions to merge, and automating that decision against a target size band, is the job of the Master's normalizer chore - covered in full in the HBase region normalizer.