HBase Balancer

Core concept: even distribution and region assignment

HBase spreads data by splitting it into regions: contiguous ranges of keys that form the unit of distribution and parallelism. Each region is assigned to exactly one RegionServer at a time, and the assignment must be tracked persistently (in Zookeeper or HDFS) so that if the server fails, the Master knows which regions to reassign. The Balancer’s job is to keep the assignment itself fair: spread regions across servers evenly so that traffic, storage, and responsibility are balanced.

A balanced cluster has multiple consequences. First, no single server is a bottleneck: when a client scans a range of keys it touches multiple servers in parallel, and the slowest server is your ceiling. Second, write throughput scales linearly: each region is a write-ahead log (WAL) on a single server, so distributing regions distributes write load. Third, storage is even: if all servers have the same hardware, imbalance means some fill up while others sit half-empty, wasting cluster capacity. A perfectly balanced cluster means every server carries roughly the same number of regions, the same amount of request traffic, and the same amount of data.

Region assignment and how balancing moves regions

A region is assigned to a RegionServer by writing its assignment state to a persistent location: traditionally the hbase:meta table, but more recently hbase:assignment in HBase 3.x+. When a region comes online (at startup or after a move), the server loads it from HDFS, opens the store files, and begins serving requests. The Balancer runs on a timer (default every 300 seconds) and asks: are the regions distributed fairly?

If the answer is no, the Balancer creates a plan: a sequence of moves. A move is an instruction to reassign a region from Server A to Server B. The Balancer does not execute the move immediately; instead it sends the plan to the Master, which executes it in the background. During a move, the region on Server A is taken offline (no reads or writes), the assignment is updated, and Server B opens the region fresh. The old blocks still live in HDFS; Server B finds them by block locality, but if the blocks were written by Server A (the hot-spot we are escaping), Server B has no local copy, so it reads them over the network. This is the key cost: moved regions lose locality until compaction rewrites the data and co-locates blocks with their new home server.

Balancing policies: four flavors

HBase ships with four main balancing policies, and the choice between them depends on what you want to balance:

The StochasticBalancer is the modern default because it juggles multiple signals at once. Instead of balancing only region count, it weighs region count, request rate (requests per region), write rate, store file size, server load, and locality cost all together, then uses a cost-function to decide if swapping two regions would improve the overall score. The algorithm runs iteratively: pick two random regions on different servers, estimate the cost before and after swapping, and accept the swap if it lowers total cost. This stochastic approach is slower than SimpleBalancer but usually produces better overall balance across multiple dimensions.

Capacity and metrics balancing

A region is more than a line in the assignment table; it carries metrics that reflect its actual load. The Master collects these from each RegionServer every few seconds: how many requests (reads + writes) the region handled, how much data it stores, how many store files it has. A perfectly balanced cluster means not only even region count but also even distribution of load.

The Stochastic Balancer uses cost functions to measure imbalance across these dimensions. For example, the request-rate function computes the coefficient of variation of requests per region, and penalizes uneven distribution: a server with one massive hot region (10,000 requests/s) and one cold region (100 requests/s) gets a higher cost than a server with two regions at 5,000 requests/s each. Similarly, the size function penalizes servers that carry unusually large or unusually small regions. The Balancer weights these functions (configurable via hbase.master.balancer.stochastic.* configs) and seeks to minimize their sum.

This multi-signal approach avoids the trap of region-count-only balancing: a cluster with 10 regions per server might be perfectly balanced by count, but if one server’s 10 regions are all hot (thousands of requests) and another server’s 10 are cold (hundreds of requests), the hot server is the bottleneck. The Stochastic Balancer detects this and moves a hot region off the hot server, even if it slightly increases imbalance by count.

Rack awareness and data locality

HDFS spreads replicas of each block across racks to survive rack-level failures. A block is typically replicated three times: two on the same rack (for speed), one on a different rack (for fault tolerance). When a RegionServer reads a block, it prefers the local copy (on-disk or in-memory), then falls back to the same rack over the network, then to a remote rack.

The Balancer is rack-aware: when it assigns or moves a region, it tries to place it on a server that has replicas of that region’s blocks on the same rack, to exploit locality. At assignment time, this is the TableInputFormat’s job: when the Master bulk-assigns regions after a restart, it uses the HDFS block map to assign each region to a server that holds its primary replicas, achieving high initial locality.

But after a region move, locality is temporarily lost. The region is now on a new server, but the blocks it reads are still in their original locations. Over time, as the server writes new data (during flushes and compactions), new blocks are created on the local server, and locality gradually recovers. A server might have 95% locality for an old, rarely-written region but 0% for a region it just received that is getting heavy writes from a remote region. The Balancer accounts for this: the LocalityCostFunction measures weighted locality across all regions and can be tuned to penalize moves that sacrifice locality.

The cost of moving: offline time and cache warmth

A region move is not instantaneous. When the Balancer decides to move region R from Server A to Server B, the sequence is:

1. Close on A: The region on Server A is marked for close and stops accepting new writes. In-flight requests finish, then the region flushes its memstore to disk and closes.
2. Update assignment: The Master writes the new assignment (B) to the persistent state.
3. Open on B: Server B opens the region: scans its blocks from HDFS, populates metadata, and begins serving requests.

Total time: typically 1–10 seconds for a small region, longer for large ones (if the memstore is big) or if HDFS is slow. During this window, clients trying to write to the region get a NotServingRegionException, retry (and get directed to the new server), and proceed. The latency spike is usually brief, but in a cluster with dozens of concurrent moves, the aggregate effect is real.

More insidious: block cache warmth. Server A has cached hot blocks in memory (the HBase block cache, typically several gigabytes). When the region moves to Server B, that cache is lost. Server B starts with a cold cache, so the first reads come from disk (or over the network), much slower than a cache hit. For a heavy read workload, this can mean 10x latency increase for hours until the new server’s cache warms. This is why aggressive balancing can hurt: the cluster becomes more evenly distributed but individual read latencies spike.

Pre-balancing and balancer throttling

Recognizing these costs, HBase provides ways to tune when balancing happens and how fast it proceeds. Pre-balancing means running the Balancer immediately after a bulk data load, when regions are often assigned unevenly. Instead of waiting for the periodic balancer (300 seconds), you invoke it manually, so it spreads the load as soon as the data lands.

Throttling is controlled by:
hbase.master.balancer.max.balancing: Maximum number of regions to move per balancing run (default 4). Lower values slow down balancing but reduce latency spikes. Higher values finish faster but risk overwhelming the cluster.
hbase.master.balancer.period: How often (in ms) the Balancer runs (default 300000 = 5 minutes). During heavy load, increase this to avoid interference with application traffic.
hbase.master.balancer.stochastic.numRegionLoadsToRemember: How many previous load samples to use for smoothing (default 15). More samples smooth out transient spikes but add latency to the cost function.

A common tuning pattern: disable the periodic balancer during peak hours (hbase.balancer.period=0) and run it manually at off-peak, or set very high max.balancing (e.g., 10+) to finish fast and take the latency hit once rather than multiple small hits.

Trade-offs and operational gotchas

The Balancer is powerful but fragile. Tune it wrong and you get either a perpetually imbalanced cluster (if the threshold for moving is too high) or constant thrashing (regions moving every cycle, cache never warm). Common mistakes:

A second theme: balancer thrashing. If the cost function is too sensitive, the Balancer can cycle: accept a move because it lowers cost by 0.01%, then the next run rejects it, then accepts it again. This shows up as regions constantly opening and closing. The fix is to increase hbase.master.balancer.stochastic.costSlack so that a move must improve cost by at least, say, 2%, not 0.01%.

Third: locality death. An aggressive balancer that ignores locality cost can scatter a region’s blocks across racks. If your cluster is bandwidth-constrained, this can hurt throughput more than perfect balance helps. The fix: weight the LocalityCostFunction heavily, or use hbase.master.balancer.stochastic.localityCostFunctionRatio (default 25%) to preserve locality.

Monitoring and diagnosis

When balancing goes wrong, several tools help diagnose. In the HBase shell:

# Check if balancer is running
hbase shell > balancer_enabled
# Run balancer immediately
hbase shell > balancer
# See region distribution by server
hbase shell > status 'detailed'

In the Master UI (typically :16010), navigate to Procedures & Locks to watch active region moves, and Master Status to see the latest balancer run stats. In logs, look for HMaster.balancer: lines like Balancer's best function = X.YZ show the cost score before and after the move proposal.

Metrics to watch:
hbase.master.cluster.metrics_region_count_stdev: Standard deviation of region counts per server. Low = balanced; high = imbalanced.
hbase.master.cluster.metrics_readreq_count_mean/stdev: Request rate; stdev relative to mean shows read-load imbalance.
hbase.master.balancer_time_in_balancer: How long the Balancer took to make a decision (ms).

Best practices: when to trust and when to intervene

The Stochastic Balancer is sophisticated, but it is not a mind reader. It responds to metrics, not intent. If your workload is truly uneven by design (you have a few hot regions that MUST stay unbalanced for business reasons), the Balancer will try to spread them out, fighting your workload. In such cases:

• Disable balancing for specific regions: Use split_region to give hot regions their own small size, then disable balancing on that table during known hot periods.
• Use table-level configuration: Set BALANCER_IGNORED_METRICS = 'true' on the table, or use hbase_group_balancer to define server groups and balance within groups.
• Manual intervention: Disable the balancer, use move_region in the shell to place regions exactly where you want them, then re-enable balancing with conservative thresholds.

General guidance: run the Balancer with default Stochastic settings, monitor region distribution and request rates, and only tune if you see clear imbalance or constant thrashing. Most clusters benefit from the default, and heavy tuning is a sign you are fighting the workload instead of accommodating it.