The HMaster is the most consistently misread process in an HBase cluster. Its name suggests a coordinator that every request passes through, and the mental model that follows — master saturated, cluster slow — is wrong in both directions. Clients read and write against RegionServers directly, and the master does no work at all for them. What the master owns is the control plane: which server holds which region, what the schema says, what happens when a server dies, and which of the thousands of half-finished multi-step operations in flight needs another nudge. That work is invisible when it is healthy and impossible to ignore when it is not. This page walks the master as a process — election, initialization, the assignment state machine, DDL, crash recovery, its chore inventory, and the specific ways it goes wrong.
Why the master is not on the read and write path
The HMaster is often mistaken for the primary controller of HBase. It behaves far more like a background coordinator. A client that wants a row resolves the row key to a region and that region to a RegionServer, then talks to that server directly; the master is not consulted, not proxied through, and not even necessarily running. That resolution happens against hbase:meta, which is itself an ordinary table hosted by an ordinary RegionServer, and the result is cached in the client for as long as it keeps working. The consequence is the single most useful fact about the process: HMaster performance almost never limits HBase throughput.
It also means a cluster survives a dead master far better than intuition suggests. With no master at all, existing regions stay open and keep serving reads and writes indefinitely. What stops is everything that changes the shape of the cluster — no new region can be assigned, a crashed RegionServer is never recovered, splits cannot complete, DDL blocks, and the balancer goes quiet. So a master outage is not an immediate outage; it is a cluster that has lost its ability to heal, and that degrades into a real outage on the first RegionServer failure. Knowing this stops two common mistakes: overinvesting in master hardware, and treating a master alert as a page-the-whole-team event when it is usually a fix-it-this-hour event.
Read the rest of this page as the inventory of what you lose while the master is down.
Election, fencing, and the backup masters
Every master process starts as a candidate. It races to create an ephemeral znode under the HBase ZooKeeper root — conventionally /hbase/master — and exactly one wins. The losers do not exit; they register themselves under /hbase/backup-masters, set a watch on the active node, and park. Nothing is replicated between them, because there is nothing in a master worth replicating: all durable state lives in HDFS under the root directory, in hbase:meta, in ZooKeeper, and in the procedure store. A backup does not shadow the active, it simply waits to re-read the same state.
The fencing rule is what makes this safe, and it is stricter than most operators expect. An active master that loses its ZooKeeper session does not carry on and attempt to reconnect — it aborts. That is deliberate: the moment its ephemeral node vanishes, a backup is entitled to claim mastership, and two processes both believing they are active would issue conflicting assignment decisions. Suicide-on-session-loss converts a potential split brain into a clean failover. It also means a long JVM pause or a ZooKeeper hiccup can kill a perfectly healthy master, which is why zookeeper.session.timeout is generous and why masters are usually given uneventful, unshared JVMs.
Backup masters cost almost nothing and are worth running two of. The failure that actually bites is the backup nobody ever tested: wrong configuration, stale classpath, or no permission to write the procedure store, discovered at the exact moment it is promoted.
From winning the election to Master is initializing
Winning the election is not becoming useful. A freshly promoted master runs a startup sequence, and the web UI reports Master is initializing for its duration — a state operators see during failover and misread as a hang.
Broadly, the sequence is: read the cluster identity and root directory; start the RPC and info servers; wait for RegionServers to check in, because assigning regions before the fleet has reported would scatter them onto whichever two servers happened to register first; then locate and, if necessary, assign hbase:meta. Meta comes first and outranks everything, because nothing else can be reasoned about until the catalog is readable — its location is published in ZooKeeper precisely so this bootstrap has a fixed starting point. With meta online, the master loads the assignment picture, reconciles what meta claims against what RegionServers actually report they are hosting, and replays the procedure store so that every operation interrupted by the previous master resumes rather than restarts. Only then does it declare itself initialized, enable the chores, and start accepting DDL.
Two operational consequences follow. A master stuck initializing is nearly always stuck on meta — unassignable, hosted by a server that will not respond, or waiting on a recovery that cannot finish. And the wait-for-servers phase is why a full cluster restart should bring RegionServers up promptly after the master rather than trickling them in over ten minutes.
Assignment as a state machine — AMv2 and the transit procedure
Region assignment is the master core job, and since HBase 2.0 it is expressed as durable state machines rather than in-heap bookkeeping. The AssignmentManager does not hold the answer in memory and hope; every change of region ownership becomes a procedure that walks a region from one state to the next — OFFLINE to OPENING to OPEN, or OPEN to CLOSING to CLOSED — and persists each transition before acting on it. The master decides the target server, dispatches an open or close to that RegionServer, and waits for the server report that confirms it; the report, not the dispatch, is what advances the state.
Two invariants make the design work. First, a region is the subject of at most one transit procedure at a time, enforced by a per-region lock the framework records durably — so a move cannot interleave with a split, and the double-assignment class of bug that plagued the pre-2.0 design is structurally excluded. Second, hbase:meta is the single authority for who owns what. ZooKeeper carried per-region assignment state in the old design and no longer does; it holds liveness and a handful of bootstrap pointers, nothing more. When a RegionServer reports hosting a region meta disagrees about, the master reconciles rather than guesses.
The framework underneath — executor threads, rollback, the lock scheduler, the MasterProcWAL store — is covered in Procedure v2; what matters here is that the master owns the decision and delegates the durability.
The catalog the master writes and everyone else reads
hbase:meta is where the master records its decisions, and the division of labour around it is easy to get backwards. The master is the only component that should be writing assignment rows into meta — it does so as part of the transit procedures above, so that the catalog and the procedure store agree even across a crash. But the master is emphatically not in the path of anyone reading meta. Clients scan it from whichever RegionServer hosts it, cache the result aggressively, and only re-resolve when a request comes back saying the region moved.
That asymmetry explains a surprising amount of behaviour. It is why clients keep working through a master outage; why meta itself is assigned before anything else at startup; and why a meta region that is unavailable is a cluster-wide event while a master that is unavailable is not. It is also why the master treats meta assignment with a higher priority than user regions throughout — recovery work that touches meta jumps the queue ahead of cosmetic region moves.
The catalog also accumulates rows the master must eventually retire: split parents whose daughters have taken over, merged regions whose sources are still referenced. Cleaning those is a chore, covered further down. The row format, the bootstrap history, and the client caching path are the subject of the hbase:meta architecture page; treat this section as the master side of that story only.
DDL is a procedure, not a blocking RPC
All schema change goes through the master, and none of it is a synchronous call that holds a connection open until the work is done. When an admin issues create, the master constructs a create-table procedure, persists it, and returns a procedure identifier; the client polls that identifier. The procedure then writes the table descriptor, creates the region directories, inserts the regions into meta as closed, and spawns one child assignment procedure per region, suspending until they all report back. Only when the last child completes does the table flip to enabled.
alter and drop follow the same shape. Altering a table takes an exclusive lock on that table so two concurrent schema changes cannot interleave, and applying a change to a live table means closing and reopening every region so it picks up the new descriptor — which is why altering a wide table is an availability event proportional to region count, not a metadata flick. Dropping decommissions the regions and moves their files to the archive rather than deleting them outright, so a snapshot or a replication peer still holding a reference does not lose data underneath it.
The practical rule this yields: pre-split at create time rather than altering later, batch schema changes into maintenance windows, and never assume a returned shell prompt means the DDL finished. It means the procedure was accepted. list_procedures tells you whether it landed.
When a RegionServer dies — the master decision leg
RegionServer liveness is a ZooKeeper ephemeral node, not a health check. Nothing inspects whether a server is serving well; only whether its session is being renewed. When the session expires the node disappears, the master is notified, and it starts a server-crash procedure for that host — the same durable state machine as everything else, which is why a master that dies mid-recovery resumes recovery instead of forgetting it.
The master decision sequence is worth knowing precisely, because it is the sequence that defines how long rows are unavailable. Mark the server dead so nothing else is assigned to it. If it was hosting meta, recover that first — everything downstream depends on the catalog. Split the dead server write-ahead logs so each region regains its unflushed edits. Then reassign the regions to live servers, each of which replays its recovered edits before it opens for traffic. In-flight transit procedures for regions on the dead host interlock with the crash procedure rather than racing it, so an operator is never asked to reconcile the two by hand.
The expensive middle step is the log split, and its mechanics — distributed split workers, recovered edits files, the sequence-id filter that skips already-flushed data — belong to WAL splitting. What the master contributes is detection, ordering, and the guarantee that the whole sequence is restartable. The cost of the final step, opening regions cold on a new host, belongs to the RegionServer.
The balancer proposes, procedures dispose
The load balancer is a master chore, not a service. It wakes on a period governed by hbase.balancer.period — five minutes by default — examines the current distribution of regions across live servers, and computes a plan: a list of region moves that would improve it. The master then executes that plan as ordinary move procedures, one region at a time, subject to the same locks and durability as any other assignment. The balancer never touches data and never moves a byte; a move is a close on one server and an open on another, with the files left where they are on HDFS.
That last point is why balancing is not free and why the balancer is conservative. A moved region is briefly offline, arrives on its new host with a cold block cache, and loses data locality until a compaction happens to rewrite its files near the new server. Aggressive rebalancing can cost more read latency than the imbalance it corrects. Operators should also know the balancer can be switched off — balance_switch false — and that it is conventionally disabled for the duration of a rolling restart or a bulk maintenance operation, then deliberately re-enabled. Forgetting the re-enable is a classic slow-burn incident: the cluster skews for weeks with nobody noticing.
What the balancer optimizes for, and how the cost functions weigh region count against locality, read load, and table isolation, is the subject of the balancer page and the StochasticLoadBalancer internals.
The rest of the chore inventory
The balancer is the famous chore, but the master runs a small fleet of them, and most master mysteries turn out to be one of these either running when it should not or not running at all.
The catalog janitor garbage-collects meta. After a split, the parent region row survives until both daughters have compacted away their references to the parent files; the janitor is what notices that and removes the parent row and directory. Disable it during an investigation and split debris accumulates in meta indefinitely. The normalizer is the balancer counterpart for region size rather than region count, merging regions that shrank and splitting ones that grew — see the normalizer page. The log and HFile cleaners retire the old write-ahead logs and archived store files that are no longer referenced; they run as plugin chains precisely because a file may still be needed by a replication peer or a snapshot, and each plugin gets a veto. hbase.master.logcleaner.ttl sets how long old logs linger before they are eligible.
A stalled replication peer is the classic cause of an old-WAL directory that grows without bound: the cleaner is working correctly and being vetoed every pass. The master also serves the cluster status the shell and the web UI render, aggregating the load reports RegionServers send it on every heartbeat.
Regions in transition — reading a master that is stuck
A region in transition is a region partway through a transit procedure. Transient RIT is normal and invisible; the symptom that matters is a region that has been in transition for minutes or hours. The master exports this directly — a count of regions in transition, a count over the alert threshold, and the age of the oldest — and the oldest-age metric is the one worth paging on, because a slowly growing count of brief transitions is healthy while a single region stuck for an hour is not.
Diagnosis starts at the master web UI, conventionally on port 16010, which lists regions in transition with their current state. From the shell, list_procedures shows the in-flight procedure tree and list_locks shows which resource each is waiting on — together they usually explain a stall in one read: a procedure blocked on a table lock held by an abandoned operation, or waiting on a RegionServer that stopped answering. The master log then names the region and the server it keeps failing to reach.
Resist the urge to reach for repair tooling first. Many stalls clear on their own once the unresponsive server is fenced. When intervention is genuinely needed, the modern tool is HBCK2, whose verbs are deliberately narrow — bypass a wedged procedure, schedule recoveries for a server the master never processed, assign or unassign a specific region. Its predecessor rewrote filesystem state; this one nudges the state machine and lets the master do the work.
Sizing and operating the master
Master hardware barely matters, and the budget saved belongs in RegionServers. The master holds no user data, serves no scans, and caches no blocks. Its heap is sized by the assignment picture and the in-flight procedure set — thousands of regions and a large fleet, not terabytes of rows — and a heap that would embarrass a RegionServer is usually generous here. What the master genuinely needs is stability: a JVM that does not pause long enough to lose its ZooKeeper session, and disks that are not shared with something that stalls.
The one piece of state with real operational weight is the procedure store. It only advances when procedures complete, so a single wedged procedure prevents cleanup and the store grows steadily. An unexplained pile of master procedure log files is not a disk problem, it is a stuck procedure wearing a disguise, and the fix is upstream. Historically, early HBase 2.x builds also allowed the master to host system regions itself; that was reverted, and for a good reason worth remembering — putting the coordinator back on a data path reintroduces exactly the coupling this architecture spent a decade removing.
Practical checklist: run at least one backup master and restart it occasionally so you know it works; alert on oldest-RIT-age and on master-initializing lasting more than a few minutes; keep the balancer switch under change control; and treat master restarts as routine, because with durable procedures they are.