Why architecture matters here
The naive intuition — count regions, divide by servers, move until equal — is wrong in a way that costs real latency, because regions are not interchangeable. One region of a cold archive table and one region absorbing 40k writes/sec are both 'one region'. A cluster with a perfectly even region count can have one RegionServer at 95% CPU with a saturated WAL and another idling. Region count is a proxy for load, and a weak one. The StochasticLoadBalancer's read-request, write-request, memstore-size and store-file cost functions exist precisely because the proxy is weak: they let the optimizer see the load itself rather than a count.
The second reason architecture matters here is that moves are not free, and their cost is asymmetric and delayed. Moving a region means flushing its memstore, closing it (the region serves nothing during the transition), reopening it on the target server, and replaying nothing — but the region's HFile blocks still live on the DataNodes where they were written. The new server reads them over the network until a major compaction rewrites them locally. So a move buys you balance today and charges you locality until the next compaction, which may be days away. This is why MoveCostFunction exists and why locality is weighted heavily: without them the optimizer would happily shuffle the cluster every five minutes, each round trading a small count improvement for a large, long-lived read penalty.
Third, this is an approximate optimizer running under a deadline, not a solver that finds the optimum. It explores a tiny fraction of an enormous search space, keeps improvements, and stops when maxSteps or maxRunningTime expires. It can and does return a plan that is merely better, not best — and if the time budget is too small relative to cluster size, it may return almost nothing at all. Knowing that the output quality is a function of the budget, and that the objective is a knob rather than a law, is exactly what separates operators who can steer the balancer from operators who conclude it is broken and switch it off — which is the single most expensive mistake in this article.
The architecture: every piece explained
Where it runs. The balancer runs on the active HMaster only, on a timer set by hbase.balancer.period (default 300000ms — five minutes). It is a planning component, not a data-path component: it never touches a client request. Each invocation is gated. If the balancer switch is off (balance_switch false, persisted in ZooKeeper and surviving master restarts), if there are regions in transition, or if any RegionServer is in the middle of a failover, the run is skipped entirely. This gating is deliberate — balancing a cluster that is already reassigning regions compounds churn — but it is also the reason a cluster can sit unbalanced for hours while a single stuck region-in-transition silently blocks every run.
The snapshot. A run begins by materializing a ClusterState: every RegionServer, every region it hosts, per-region read and write request counts, memstore size, store file size and count, and the HDFS block locality of each region relative to each candidate server. That last piece is the expensive one and the reason the balancer keeps a locality cache. The whole optimization then happens over this in-memory model — the balancer simulates moves against the snapshot, never against the live cluster, which is what makes a million-step search affordable at all.
The cost ensemble. Each cost function returns a normalized value in [0,1] and carries a configurable multiplier; total cost is the weighted sum. RegionCountSkewCostFunction (default weight 500) measures deviation in region count per server. ServerLocalityCostFunction (weight 25) measures how much of each region's data is not local to its host. ReadRequestCostFunction and WriteRequestCostFunction (weight 5 each) measure actual served load skew. MemStoreSizeCostFunction and StoreFileCostFunction (5 each) capture write pressure and data volume. TableSkewCostFunction (35) keeps a single table's regions from clustering on one server. MoveCostFunction (7) penalizes the plan for its own size, damping churn. And the replica functions — RegionReplicaHostCostFunction and RegionReplicaRackCostFunction, weighted very heavily — enforce that read replicas of the same region never share a host or rack, because co-located replicas are not replicas.
Candidate generators. The optimizer does not enumerate moves; it samples them, and the sampler is where domain knowledge lives. RandomCandidateGenerator proposes an arbitrary move or swap. LoadCandidateGenerator biases toward moving regions off the most-loaded servers. LocalityBasedCandidateGenerator proposes moves that improve locality — typically moving a region toward the server already holding its blocks. The replica generators propose moves that fix co-located replicas. On each step one generator is picked and asked for one candidate, which is why a run with a pathological locality problem still makes progress: the locality generator keeps proposing the fix.
The loop and the plan. Each step applies a candidate to the snapshot, recomputes total cost incrementally, and accepts it if cost decreased — otherwise reverts. The loop ends at hbase.master.balancer.stochastic.maxSteps (default 1,000,000) or maxRunningTime (default 30s), whichever comes first; in practice on a large cluster the clock wins. If the final cost improved on the initial cost by more than the minimum threshold, the difference between the initial and final layouts is emitted as a list of RegionPlan objects (region, source server, destination server). Those go to the AssignmentManager, which executes each as a TransitRegionStateProcedure under Procedure v2 — a durable state machine written to the master's procedure WAL, so a master failover mid-balance resumes the moves rather than losing or duplicating them.
End-to-end flow
Follow one run on a cluster that just lost a RegionServer. RS4 dies; ZooKeeper's session for it expires; the master's ServerCrashProcedure takes over, splits RS4's WAL, and reassigns its 180 regions. Crucially it reassigns them fast, not well — the priority is availability, so those regions land wherever there is room. The cluster is now live but lumpy: RS1 through RS3 carry an extra 60 regions each, and every one of those regions has near-zero locality on its new host because its HFile blocks are still sitting on the DataNodes that served RS4.
Five minutes later the balancer timer fires. First the gate: is balance_switch on, are there regions in transition, is a server still in failover? The ServerCrashProcedure has finished, RIT is zero, so the run proceeds. It builds the snapshot — 4 remaining servers, ~740 regions, per-region loads, and the locality matrix that now contains a large block of zeros for the orphaned regions. It computes initial total cost, and that cost is high from two directions at once: region count skew (three servers overloaded, one replacement server empty after the operator restarted RS4) and locality (60 regions reading every block over the network).
Now the loop runs. Step 1: LoadCandidateGenerator proposes moving a hot region off RS1 onto the fresh RS4. Rescoring finds region-count skew improved and locality unchanged (the region has no locality anywhere useful), so total cost drops and the move is kept. Step 2: RandomCandidateGenerator proposes swapping two cold regions between RS2 and RS3 — count skew unchanged, locality worse, MoveCostFunction up because the plan grew. Cost rises, the candidate is reverted. Step 3: LocalityBasedCandidateGenerator proposes moving a region back toward the DataNode holding its blocks. Locality improves sharply, count skew degrades slightly; the weighted sum drops, so it survives. This is the entire algorithm — thousands of these micro-decisions per second, each one a small argument between cost functions, settled by the weights.
Thirty seconds later maxRunningTime expires, typically after a few hundred thousand steps. The optimizer has explored a vanishing fraction of the possible layouts and has no idea whether it found the optimum — it only knows the final layout costs less than the initial one. It diffs the two and emits, say, 96 RegionPlans. The AssignmentManager begins executing them, throttled: each plan becomes a TransitRegionStateProcedure that tells the source RS to close the region (flushing its memstore first — this is where a big write-heavy region takes seconds, not milliseconds), waits for the close to be acknowledged, updates hbase:meta, then tells the destination RS to open it. Between close and open, that region is unavailable and clients retry; this is normal, brief, and the reason you do not want a 3000-plan balance at peak hour.
The important part is what happens after. The cluster is now count-balanced, and its aggregate locality is worse than before RS4 died — those 96 moved regions read over the network. Latency is measurably higher. Nothing is broken; the balancer made a deliberate trade. Locality recovers only when a major compaction rewrites each region's HFiles, at which point HDFS writes the new blocks locally to the current host. If your major compactions are scheduled weekly, you live with degraded reads for a week — which is why operators often trigger a major compaction after a large rebalance instead of waiting.
Five minutes later the timer fires again. The balancer rebuilds the snapshot, recomputes cost, and finds the improvement now available is below the minimum threshold. It emits no plans and logs that the cluster is balanced. It will keep doing this — cheaply, every five minutes, forever — until the next split, crash, or traffic shift makes a better layout available. The steady state of a healthy balancer is doing nothing, loudly enough that you can tell it is still watching.