Why architecture matters here
The pre-2.0 control plane failed for an architectural reason worth internalizing: multi-step operations held their progress in the master’s heap, coordinated through ZooKeeper flags that were neither atomic with meta updates nor expressive enough to describe where in a sequence the operation stood. When the master died — or worse, when two components raced (assignment retries against a split, a disable against a merge) — the cluster’s state became the union of partial writes across HDFS, meta, and ZK, and no component owned reconciliation. The infamous outcomes: double-assigned regions (two RegionServers serving the same rows — data corruption, not just unavailability), orphan regions on disk absent from meta, and RITs that only a human with hbck -fix* flags could clear.
Procedure v2 matters because it moves the invariant from ‘operators repair inconsistency’ to ‘the system cannot forget what it was doing.’ Write-ahead persistence of every state transition gives crash-restart the same guarantee the data path has had forever: replay to the exact point of interruption. Explicit rollback paths make failed operations unwind cleanly instead of abandoning debris. And per-resource locking inside the framework serializes the races — a region can be the subject of exactly one transit procedure at a time, a table’s DDL queues behind its exclusive lock.
The operational payoff is direct: assignment storms after master failover became deterministic replays; hbck1’s invasive repair surgery was retired for HBCK2’s narrow ‘nudge a stuck procedure’ verbs; and the master restart, once a cluster-wide event with hours of RIT fallout, became routine. Understanding the framework is now the difference between reading list_procedures output fluently during an incident and staring at it.
The architecture: every piece explained
Top row: submission and execution. Admin calls (create_table, move_region, split) construct a Procedure object — a state machine with an execute() returning either more work, spawned children, or completion, and a rollback() per state. The ProcedureExecutor in the active master runs a pool of worker threads that pull runnable procedures from the scheduler and drive one state transition at a time. The iron rule: before any transition’s effects are executed, the transition is persisted to the procedure store — historically the MasterProcWALs directory of write-ahead logs, replaced in later 2.x by a region-based store (a local, non-replicated HBase region) that eliminated WAL-cleanup bugs. Persist, then act; never the reverse.
Middle row: composition and safety. Procedures form trees: CreateTableProcedure spawns per-region assign children and suspends until they report; ServerCrashProcedure spawns WAL-split then region-reassign children. Suspension is cheap — a waiting procedure occupies no thread. The lock scheduler enforces isolation: exclusive table locks for DDL, shared for region-level work, per-region exclusivity so two moves cannot interleave; locks are themselves recorded so they survive failover. AssignmentManager v2 expresses every region state change as a TransitRegionStateProcedure (TRSP) walking OPENING → OPEN or CLOSING → CLOSED, updating hbase:meta as the single authority — ZooKeeper no longer carries assignment state, ending an entire class of meta/ZK divergence.
Bottom row: the remote leg. Steps that need a RegionServer — open this region, close that one — go through a remote dispatcher that batches ExecuteProcedures RPCs to the target server; the RS executes and reports back, and the report is what advances the TRSP’s state. If the RS dies instead, its ServerCrashProcedure will interlock with the orphaned TRSPs — the framework, not an operator, owns that reconciliation.
The framework imposes a discipline on procedure authors that explains its reliability: every execute() step must be idempotent, because crash-replay may re-run the last persisted-but-unconfirmed step; side effects must be verifiable (create-if-absent, open-and-check) rather than blind. Procedures carry priorities — meta region assignment outranks user regions, ServerCrashProcedures outrank cosmetic moves — so recovery work jumps DDL queues. And long waits are modeled as events, not polling: a procedure waiting on a child, a lock, or a region server report parks itself and is re-queued only when the event fires, which is why a master can carry tens of thousands of suspended procedures with a worker pool of sixteen threads and negligible CPU. The design is a small workflow engine, specialized for exactly one tenant.
End-to-end flow
Follow create 'orders', {NUMREGIONS => 64}. The master constructs a CreateTableProcedure, persists its initial state, and returns a procedure ID to the client (which polls for completion). Workers drive it through its states: write table descriptor to HDFS, create the 64 region directories, insert the regions into hbase:meta as CLOSED, then spawn 64 child TRSPs and suspend. Each TRSP acquires its region lock, persists OPENING, asks the balancer for a target server, and dispatches a remote open. As RegionServers confirm opens, each child updates meta to OPEN, completes, and decrements the parent’s wait count; the last child wakes the parent, which flips the table to ENABLED and completes. Every arrow in that sentence hit the procedure store first.
Now the interesting run: the master is killed after 40 of 64 regions opened. The standby wins mastership, replays the procedure store, and reconstructs the exact tree — parent suspended, 40 children complete, 24 in various states. Workers resume the 24: those persisted as OPENING but unconfirmed re-verify against the target RS (idempotent — the RS reports the region already open, or the open is re-dispatched); untouched ones proceed normally. The client’s poll, retried against the new master, finds the same procedure ID and eventually the same success. No RIT pileup, no half-created table, no hbck.
Contrast a failure requiring rollback: CreateTableProcedure discovers the namespace quota is exceeded at step three. The framework drives rollback() in reverse state order — delete region dirs, remove the descriptor — each rollback step itself persisted, so a crash mid-rollback resumes the rollback. The operation’s contract is all-or-nothing as far as cluster state is concerned, which is precisely what the old control plane could never promise.
The same machinery handles the ugliest case HBase has: ServerCrashProcedure. A RegionServer’s ZooKeeper session expires; the master enqueues an SCP for it at high priority. The SCP’s states: mark the server dead, split its WALs (child procedures per WAL, farmed to healthy RegionServers), then spawn TRSPs to reassign every region it carried — each of which replays recovered edits before opening. If a TRSP for one of those regions was already in flight when the server died (say, a move that had reached OPENING on the now-dead server), the SCP and the TRSP interlock through region state and locks: the transit is failed over to a new target rather than abandoned. This choreography — dozens of procedures, hundreds of persisted transitions, zero operator actions — is what an RS crash costs now, and every step of it is inspectable in list_procedures while it runs.