HBase Coprocessors — Server-Side Code for Triggers and Custom RPCs

What a coprocessor actually is

A coprocessor is a class you compile, package into a jar, and arrange for HBase to load into a server process. From that moment HBase treats your code as part of itself. It lives in the server's heap, runs on the server's threads, is reached through the server's classloaders, and shares the server's fate. Nothing about the mechanism is remote, asynchronous or optional once it is installed - it is an extension point that lets you edit HBase's behaviour without forking it.

There are two families, and the difference between them is not cosmetic. An observer is a trigger: you never call it, HBase calls it, at fixed points inside operations the server was going to perform anyway. An endpoint reverses the direction of control: it is a procedure installed on the server that a client deliberately invokes, the closest thing HBase has to a stored procedure.

Choosing the wrong family is the first mistake people make. Teams reach for an endpoint when what they actually needed was an invariant that holds whether or not the writer cooperates, or they bury a rewrite in an observer when the application wanted an explicit call it could see in a stack trace. Observers are for rules that must hold for every client, including the one written by somebody who has never heard of your rule. Endpoints are for computation you want to move to where the data already is.

Observers: hooks in the server's own code paths

Observer methods come in pre/post pairs wrapped around specific points in code the server already runs. On the write path there are hooks around the whole batched mutation, around the individual Put and Delete, and around the read-modify-write operations - Increment, Append and the check-and-mutate family - which matter separately because those already hold a row lock when your code runs, so anything slow inside them holds the lock too.

On the read path there are hooks around the Get and around the scanner lifecycle: open, next, close. Scanner hooks are the ones with real reach, because they let you substitute or wrap the scanner itself and therefore change what a scan can see rather than merely inspecting what it returned.

The third group is the region lifecycle, and it is the one people forget exists. There are hooks around region open and close, around flush, and around compaction - including the point where the server hands you the scanner it is about to use to merge store files. That is how retention, redaction and derived-column behaviours get implemented without touching the read path at all, and it is also where the nastiest coprocessor bugs live, because a hook that only fires during compaction can sit undisturbed for a week before it takes a server down.

Hooks chain. If several coprocessors register for the same point they are invoked in priority order, with system-priority implementations ahead of user-priority ones, and each sees whatever the previous one left behind. Method names have shifted across major versions and the interfaces themselves were reshaped in the 2.x line, so write against the javadoc of the release you actually run rather than against a code sample of unknown vintage.

For a step-by-step trace of one mutation moving through the registered hooks in order, and of an aggregation call fanning out across regions, see the coprocessor architecture walkthrough. This page deliberately does not repeat that sequence; it is about what the machinery costs you once it is installed.

What a hook is allowed to do: modify, veto, augment

Modify. The object handed to a pre-hook is the live one. Adding, removing or rewriting cells on a Put before the core operation runs changes what is durably written; there is no copy and no later reconciliation. This is how server-side defaulting, normalisation and tagging are implemented, and it is also why a careless hook can make the data on disk disagree with what the client believes it sent.

Veto. Throw from a pre-hook and the operation fails, with your exception surfaced to the client. This is the entire basis of server-side authorisation: the check runs before the mutation, and refusing is simply not returning normally. Note the asymmetry with post-hooks. Throwing from a post-hook produces a client-visible error for a write that has already been appended to the log and applied to memory. The client sees a failure, retries, and now you have the write twice. If a post-hook can fail, it must swallow its own failures or you have built a duplicate generator.

Augment. Some hooks let you supply the answer and tell the framework to skip its own default handling. This is by far the sharpest of the three, because HBase's default handling includes bookkeeping you may not know about, and later 2.x releases deliberately narrowed the set of hooks that honour a bypass at all. Treat it as a last resort rather than a design pattern.

Across all three, decide up front whether a failure inside your coprocessor should fail the user's operation or be logged and ignored, and write that decision down next to the code. An authorisation hook must fail closed. An audit hook must fail open. An index maintainer sits between the two and deserves an actual argument rather than whatever the exception handling happened to do.

RegionObserver, MasterObserver, WALObserver - three blast radii

The three common observer types differ far more in consequence than in interface, and picking the narrowest one that can see what you need is most of the safety work.

RegionObserver

Hooks the data path for regions of the tables it is attached to. It is invoked once per operation - millions of times a day on a busy table - so it is the type where cost per invocation dominates, and the type where a table-level attachment keeps the damage scoped to one table's regions rather than to a whole cluster.

MasterObserver

Runs in the HMaster and fires on administrative work: table create, delete and modify, region assignment, balancer runs, snapshot operations. Invocation frequency is trivially low, which makes it feel harmless, and the blast radius is what makes it not. A dead master leaves existing reads and writes running - clients route through cached region locations - but nothing can be created, assigned, split or recovered until it comes back. Worse, a standby master reads the same configuration, so a master coprocessor that aborts the process on startup takes the standby with it. Failover is no protection against a fault that is in the configuration rather than in the machine.

WALObserver

Sits on the write-ahead log, which means it sees every mutation on that server for every table, whether or not that table wanted a coprocessor. Time spent there is time added to log append and sync, the most serialised part of the write path, so a WALObserver is the one place where a slow hook degrades the durability path itself. See WAL durability for why that segment is so unforgiving.

There is also a server-level observer for events such as region merges and replication endpoint lifecycle, useful for a narrow set of infrastructure work and rarely the right answer for application logic.

Endpoints: RPC you call, and the aggregation case

An endpoint is defined as a protobuf service. You generate the stubs, implement the service in a class the server loads, and the RegionServer exposes it as an additional RPC method alongside the built-in ones. The client invokes it over a key range.

The detail that trips people up is that one client call is not one server call. The invocation fans out to every region that overlaps the range and produces one result per region, and merging those results is the client's job. There is no coordination between regions and no transaction across them. Anything whose correct answer requires seeing two regions at the same instant cannot be built this way, and the fact that it usually works in a small test table with one region is precisely how that bug reaches production.

Aggregation is the case that motivated the feature and is still the best example. Counting or summing a column over a hundred-million-row table from the client means shipping a hundred million rows across the network so the client can add one to a counter each time; almost all of that traffic exists only to be discarded. An endpoint scans locally on each region and returns a number per region, so the network carries one small message per region regardless of table size. HBase ships an aggregation implementation and a matching client for count, sum, average, min, max and standard deviation, with the caveat that HBase stores bytes and has no idea they are numbers - you supply an interpreter that tells it how to decode the column. Phoenix generalises the same idea, compiling SQL aggregation and joins into endpoint calls.

Before writing an endpoint, ask whether a filter would do instead. Filters also run server-side, need no jar deployment and no restart, and cost nothing when you stop using them. The dividing line is simple: a filter reduces what gets shipped, an endpoint replaces what gets computed. And remember that an endpoint invocation occupies an RPC handler for its entire duration, so a long scan inside one is indistinguishable, from the server's point of view, from a long client scan - see the RegionServer's handler and memory model for what that costs under concurrency.

There is no sandbox

This is the single most important fact about coprocessors, and everything operational follows from it. Your code does not run in a container, a child process, or a restricted security context. There is no memory limit, no CPU limit, no wall-clock timeout, and no supervision. It runs inside the RegionServer JVM with the server's full permissions on the server's heap.

So an unbounded allocation in your hook is an unbounded allocation in the RegionServer, and the JVM that dies is the one hosting regions for every table on that machine - not just the table your coprocessor was attached to. An infinite loop parks an RPC handler permanently, and enough of them starve the pool until the server stops answering anything. Recovery is not instant either: the regions have to be reassigned and the dead server's log has to be split and replayed before those regions serve reads again.

The worst shape this takes is the poison region. If the fault is deterministic and triggered at region open - a hook on the open path, or a failure while the coprocessor itself is being instantiated - then the region cannot be opened anywhere. The master assigns it, the receiving server fails, the master assigns it somewhere else, and one bad region walks the cluster taking servers down in sequence while the assignment stays permanently in transition. Nothing about this is exotic; it is the ordinary outcome of a null pointer in a hook that reads region metadata.

HBase's configured default is to abort the server when a coprocessor errors, governed by hbase.coprocessor.abortonerror. That reads as hostile until you consider the alternative: a server that silently kept serving after your authorisation coprocessor stopped running. Failing loudly is the correct default for a mechanism with no isolation.

The lever worth knowing about before you need it is the cluster-level switch that disables user (table-level) coprocessors, hbase.coprocessor.user.enabled. Setting it false and restarting brings servers back up ignoring every coprocessor attached to a table descriptor, without your having to edit descriptors on a cluster that will not stay up long enough to run an alter. Find it in your version's configuration before the incident, not during it.

Static loading, dynamic loading, and the classpath problem

Static registration lives in hbase-site.xml, in the properties that list region, master and WAL coprocessor classes, and the jar must be on the server's classpath at process start. The scope is everything: every server, every table, including the system tables. Changing the list is a rolling restart of the cluster. This is the right mechanism for genuine cluster policy - HBase's own security implementations are configured this way - and the wrong mechanism for anything owned by one application team, because it makes one team's deploy into a cluster-wide restart.

Dynamic or table-level registration puts the coprocessor in the table descriptor as an attribute naming a jar path in HDFS, the class, a priority and optional arguments. It is set from the shell and takes effect as regions reopen, which disabling and re-enabling the table forces.

disable 'events'
alter 'events', METHOD => 'table_att',
  'coprocessor' => 'hdfs:///apps/hbase/cp/idx-2.4.1.jar|com.example.IndexObserver|1073741823|maxRetries=2'
enable 'events'
describe 'events'      # confirm the attribute landed

Scope is one table, which is what you almost always want, and removal is another alter rather than a restart. Note the version in the jar filename: replacing a jar at the same path does not reliably force a reload, because the loaded classloader is keyed by that path. Version the filename and the problem disappears.

Then there is the classpath. Your jar's transitive dependencies are resolved next to the server's own. Table-level jars get their own classloader, but HBase's classes come from the parent, so you cannot ship a different version of anything the server also uses - a jar that drags in its own copy of a common utility library is the classic way to produce a linkage error that appears only when a region opens on a server running a slightly different build. Shade what you must bring, bring as little as possible, and depend only on the published coprocessor API.

Versioning is the same problem on a longer timescale. Coprocessor interfaces are among the most exposed surfaces HBase has, and they changed shape across the 1.x to 2.x boundary. The practical consequence is that a coprocessor becomes a hard dependency of your upgrade plan: every major version bump means recompiling it, retesting it, and having somebody who understands it available. A coprocessor whose author left the company is one of the more common reasons a cluster is still on an old release.

Why a slow coprocessor becomes a latency floor

Whatever a hook costs, every operation routed through that hook pays it, forever, with no way for a caller to opt out. Two milliseconds of work in a write hook makes every write to that table two milliseconds slower, and because the cost is incurred on the handler thread that is already holding the request, it converts directly into reduced throughput as well as increased latency. The RegionServer page covers why handler occupancy, not CPU, is usually the binding constraint.

The mean is rarely what hurts. A hook that is fast except when a lookup misses its cache gives the whole table a tail shaped like that miss path, and since the hook fires on every operation there is no sampling and no amortisation - the p99 of the hook is very nearly the p99 floor of the table.

The genuinely dangerous version is a coprocessor that performs its own RPC. A write hook that reads or writes a second HBase table opens a client connection from inside the server, and the write path of the first table now depends on the availability, the region locations and the handler pool of the second. When both tables live on the same servers, which they will, you can arrive at a server whose handlers are all blocked inside hooks waiting on RPCs that need handlers on that same server. It deadlocks under precisely the load the feature was built for. If a hook truly must touch another table, give it a short timeout and treat that timeout firing as a normal, handled outcome rather than an exception.

Measure before you widen. RegionServers publish per-coprocessor execution time through their JMX metrics; put that on the same dashboard as request latency and handler queue time before the rollout, because after an incident you will be trying to prove which of several simultaneous changes moved the graph.

What coprocessors are genuinely good for

The strongest argument for the mechanism is that HBase builds on it itself. Server-side access control and cell-level visibility labels ship as coprocessors, and the RegionServer grouping feature is implemented as a master coprocessor - see RSGroups. A project does not put its own security model on an extension point it considers unreliable.

Secondary index maintenance is the canonical third-party use, and it is how SQL layers keep indexes in step with base tables: a write hook derives the index rows and writes them alongside the primary mutation. See secondary indexes for the design space. There is one trap worth spelling out, because it costs real correctness: the index exists only because the mutation hooks fire. A bulk load hands the server files that are already in HFile format and the server moves them into place, so the mutation path is never entered and your hook is never invoked. Rows appear in the table; nothing is derived from them. Nothing errors, no counter moves, no log line is written, and the divergence surfaces weeks later as a lookup that finds fewer matches than it should with no signal that anything is absent. Any table carrying a coprocessor-maintained index needs a stated answer for how it gets loaded - and "only ever through the API" is a fine answer, as long as somebody actually decided it.

Aggregation pushdown is the endpoint case above: correct, bounded, and a large constant-factor win on network traffic.

Enforcement at the door is underrated. Rejecting a write whose row key does not match the agreed layout, or whose required column is missing, costs one comparison per operation and saves an eventual cleanup job over a table that nobody can now safely scan. A rule enforced where the data enters is worth several rules enforced in a pipeline that runs afterwards.

Audit logging of who touched what, done as a fail-open post-hook, is cheap and hard to bypass, which is the whole point of doing it on the server.

The common regrets

Business logic. Rules that change with the business do not belong in a jar loaded by a database. Every rule change becomes a jar build, an HDFS upload and a region reopen, and the rule is invisible to anyone reading the application that appears to be writing the data.

Calls to external services. A hook that makes an HTTP request has coupled the availability of a storage cluster to the availability of something with a deploy cycle of its own. Under a partial outage the failure mode is not an error, it is every handler on every server sitting in a socket read.

Anything needing atomicity across regions. HBase guarantees atomicity for a single row. An observer that writes an index row in a different region is performing a second, independent write that can fail after the first succeeded, and no amount of careful exception handling changes that. This is the actual defect in most hand-rolled index coprocessors: they drift, quietly, and the rebuild job that reconciles them becomes a permanent fixture.

Papering over a schema mistake. An observer that rewrites keys on the way in to compensate for a row key design that hot-spots is a permanent tax paid on every operation to avoid a one-time migration. Fix the key design instead.

Governance on a shared cluster. Table-level loading means anyone who can alter a table descriptor can load arbitrary code into servers that host other teams' data. That is a review process problem rather than a technical one, but it is a real one, and the answer is to decide deliberately whether user coprocessors are permitted at all on a multi-tenant cluster.

Testing and rolling one out safely

Test against a real mini cluster with the coprocessor actually loaded on a real table, not against the hook method called directly from a unit test. Almost every interesting failure is in the interaction with the framework - what the server passes you, when it calls you, what it does with what you return - and a direct method call exercises none of it.

Drive the whole region lifecycle in that test, not just a write. Open, flush, compact, split, close, reopen. Bugs in the open hook and the compaction hook are the ones that reach production, precisely because the write path test that everyone writes never touches them.

Feed it hostile input deliberately: an empty value, a multi-megabyte cell, a row missing the column your code assumes exists, a Delete where you expected a Put, a batch of ten thousand mutations arriving in a single RPC, and the same row written twice in that one batch. Your hook will meet all of these on its first day.

Bound everything it does. No collection that grows with the number of rows seen. No per-cell object allocation you can avoid, because allocation rate in a hook becomes GC pressure on a heap that is already tuned for the cache and the memstore. A timeout on anything external. And an explicit, documented decision about failing open versus failing closed.

Then roll it out narrowly. Table-level attachment on a low-traffic table first, then one table that matters, watching per-coprocessor execution time, request p99, handler queue time and GC pause distribution at each step. Widen table by table. Never introduce a coprocessor in the same change window as a version upgrade or a configuration change - if latency moves, you need to know which change moved it.

Finally, rehearse the removal before you need it. There are two rollbacks and they apply in different situations: an alter that drops the table attribute, for when the cluster is healthy, and the cluster-level switch that disables user coprocessors entirely, for when servers are aborting too fast to run an alter at all. Knowing which one you would reach for, and having run both once, is the difference between a ten-minute recovery and an unplanned architecture review at three in the morning.