HBase Overview — Wide-Column Store on HDFS for Random Access at Scale

What HBase actually is, and how to read this page

HBase is an open-source implementation of the Bigtable design: a distributed, persistent, sorted map from byte-string keys to byte-string values, running on top of a filesystem that cannot update a file in place. That last clause is the whole engineering problem. HDFS gives you cheap replicated storage and excellent sequential throughput, and in exchange offers no random writes at all - a file is written once and then it is finished. HBase's job is to present random reads, single-row updates and short range scans over that substrate without ever editing a byte that a reader might already hold open.

Almost everything peculiar about HBase falls out of that constraint. Updates become new versions rather than overwrites. Deletes become markers that hide older cells until some later rewrite physically drops them. Files accumulate, so something has to periodically fold them back together. A single point lookup has to merge several sources to be sure it found the newest value. None of that is incidental complexity; it is what random access costs when the storage layer underneath only does append and rename.

This page is the map. Each section covers one layer at the same altitude and then hands off to the article that goes deep on it, so nothing here is the last word on anything and every section names its successor.

Why it matters

Analytics workloads often need to combine batch-scale storage with operational access patterns. A billion-user profile store that also feeds ML training pipelines is a natural HBase workload: massive size, key-based access from serving traffic, and range scans from batch pipelines. Solving this with two separate systems doubles the ops burden and creates consistency problems.

HBase also plays well with the Hadoop ecosystem. Spark and MapReduce can read HBase directly. Phoenix adds SQL. This ecosystem alignment matters when you already have a Hadoop investment.

The data model is a sorted map, not a table

A cell is addressed by four coordinates: the row key, the column family, the qualifier (the column name), and a timestamp. Supply all four and you get bytes back. HBase neither knows nor cares what those bytes mean - there is no server-side type system, and the client library that turned your integer into four bytes is the only thing enforcing the encoding.

Two of those coordinates behave nothing like each other. Column families are declared when the table is created and are a physical decision: each family has its own in-memory buffer, its own set of files on disk, and its own compaction schedule, which is what makes a family the boundary along which storage, caching and expiry can be tuned independently - and also why two or three families is usually the practical ceiling. Column family design is where that trade-off gets worked out. Qualifiers, by contrast, are not declared at all. They are arbitrary bytes chosen at write time and stored with every cell, so one row may carry a single qualifier and the next row a hundred thousand entirely different ones.

That is what sparse means here: a row without a given column does not store a null, it stores nothing and costs nothing, which is why a table with a million possible columns and five populated per row is a perfectly ordinary HBase table. It is also why table is a misleading word for the thing. There is no fixed column list, no declared types, no joins, no foreign keys, no cross-row constraints, and exactly one index - the row key itself. Timestamps supply a third dimension: HBase retains a configurable number of versions per cell and returns the newest by default, so an overwrite is really an insert and the previous value remains readable until a version limit, a cell TTL or a compaction removes it.

The architecture

The HBase data model is a sparse, multidimensional, sorted map. A row is identified by a byte-string row key. Each row has one or more column families, and each family holds arbitrary columns (called qualifiers). Each cell — the intersection of row key and column — can hold multiple versions of a value, each timestamped.

Rows are sorted lexicographically by row key across the cluster. This global sort order is what makes range scans efficient: give me all rows starting with 'user123' and HBase can find them in log time. It is also why row-key design is the single most important schema decision in HBase.

The row key is the schema

Rows sort lexicographically by the raw bytes of the row key, globally across the cluster, and that one sort order does three jobs simultaneously. It is the only index, so any access that is not a full table scan must be expressible as a prefix or a start/stop range over the key. It is the physical layout, so rows that sort adjacent are stored adjacent and are read back in one sequential pass. And it is the routing key, because the cluster carves that same sorted space into contiguous ranges and each range is owned by one server.

The working consequence is that a row key is not an identifier. It is a composite of the fields you intend to query by, in the order you intend to query them, and permuting the fields changes what the table is capable of:

device0417|1754500000123      one device, in time order      -> prefix scan, one region
1754500000123|device0417      all devices at one instant     -> every write lands in the last region
3f|device0417|1754500000123   salted device+time             -> spread across buckets, scans fan out

Three arrangements of the same three values, and three different tables in every way that matters. Only the first answers "the last hour for device 417" with a single sequential read; only the second keeps a global time window contiguous, and it pays for that with the worst write distribution available; the third trades scan locality for write spread. There is no arrangement that is good at all of it, which is exactly why the choice is a design decision rather than a default.

Three details catch people out. Comparison is bytewise over unsigned bytes, so a variable-width numeric component does not sort numerically unless it is zero-padded to fixed width, and Java's signed integers do not sort correctly at all unless the sign bit is flipped first. Every cell physically carries a full copy of the row key, the family and the qualifier, so a 200-byte key attached to twenty short values spends more space on coordinates than on data - short keys and one-character qualifier names are a real optimisation, and the HFile format shows both what that looks like on disk and how block encoding claws some of it back. And whether to model an entity as more columns in one row or more rows with fewer columns is a genuine fork with different scan, atomicity and size properties: see wide versus tall tables.

The failure everyone meets first is that a monotonically increasing key - a timestamp, a sequence number, an auto-increment id - always sorts at the end of the space, so every write in the cluster arrives at whichever server owns the last range while every other server idles. The mitigations, and the scan cost each one imposes, are the subject of hotspotting and key distribution. What belongs on the map is only this: the key decides it, and the key is fixed for the life of the table. Changing it means rewriting every row.

Regions, RegionServers, and the master

A region is a contiguous slice of the sorted key space, and it is served by exactly one RegionServer at a time. That is ownership, not replication: HBase does not keep several live copies of a region answering reads independently. Redundancy comes from below, where HDFS replicates every file three times; availability of a particular key range comes from reassigning its region to a surviving server, not from a peer that was already serving it. Holding those two ideas apart explains most of HBase's behaviour under failure.

Regions grow with the data and split when a store file crosses a threshold - hbase.hregion.max.filesize, ten gigabytes by default, though the default policy splits well below that while a table still has few regions, so a young table divides eagerly and then settles. The split itself is a metadata operation: the two daughters begin as references into the parent's existing files and only become independent as compaction rewrites them, which is why splitting appears instant and the I/O shows up afterwards (split mechanics). A server typically hosts tens to low hundreds of regions, and the count matters because write buffers are allocated per region per family, so region count converts directly into memory pressure.

Three coordination pieces sit above all that. ZooKeeper holds an ephemeral session for every RegionServer - losing the session is how death is detected - elects the active master, and records where the catalog lives. The HMaster assigns regions, executes DDL, and drives the balancer; it is not on the read or write path, so a cluster keeps serving through a master outage right up until something needs assigning. hbase:meta is an ordinary HBase table whose rows map key ranges to servers: a client reads it once, caches the result, routes subsequent requests straight to the owning RegionServer, and re-resolves only when a request comes back saying the region has moved.

Every one of those has its own article and this paragraph is only the index to them. The four-component picture as a whole is HBase architecture; the master's own responsibilities and failover are HMaster, and what the cluster actually keeps in ZooKeeper is its own subject. hbase:meta covers routing and the client cache in full; RegionServer internals covers what a server does with the regions it holds; region placement is the balancer and its cost-function implementation; region sizing over time is the normalizer; hard isolation between tenants is RegionServer groups. When a server dies, WAL splitting is what makes reassignment safe, and region replicas is the opt-in exception to single ownership.

HBase on HDFS, and what that inherits

It is worth being concrete about the layering, because a surprising number of HBase problems turn out to be HDFS problems wearing a different hat. Under the HBase root directory, each table is a directory, each region a directory beneath it, each column family a directory beneath that, and the files inside are the store files that family has flushed and compacted. The write-ahead logs live in a separate tree, organised per server rather than per region. All of it is ordinary HDFS data, which is how HBase gets three-way replication, block checksums and rack awareness without implementing any of them, and why capacity planning for an HBase cluster is really an HDFS conversation.

The inheritance runs in the other direction too. A slow or failing DataNode in a write pipeline surfaces as latency on every put whose log append happens to touch it, so a single degraded disk becomes a cluster-visible p99. Filling the filesystem stops HBase accepting writes entirely. And data locality - the property that a RegionServer usually reads its blocks from the DataNode on the same physical machine - is a performance assumption rather than a guarantee. When a region is reassigned after a failure or a balancer decision, its files stay exactly where they were, and the new host reads them across the network until compaction happens to rewrite them locally. That deferred cost is a real and frequently missed part of what moving regions around actually costs.

The write path is cheap because the read path pays

A write is acknowledged once two things have happened: the edit is appended to a write-ahead log on HDFS, and it is inserted into an in-memory sorted structure for its region and family. Both are cheap, and neither requires finding where the row belongs among the data already on disk. When memory fills, the sorted contents stream out as one new immutable file. Random writes have been converted into sequential ones, which is the entire reason HBase's ingest numbers look the way they do, and it is the same log-structured merge design that Cassandra, RocksDB and LevelDB all run on. The mechanics, the flush triggers and the backpressure that guards them are the write path; whether the log sync sits on the critical path of every put is the knob described in WAL durability levels.

The bill arrives on the read side. A Get for one row may have to consult the in-memory buffer plus every file the region has accumulated, because any of them could hold a newer version and there is no way to know without looking. Three mechanisms keep that from being as expensive as it sounds: a bloom filter per file lets the server skip files that certainly do not contain the row, a multi-level block index inside each file turns "find this key" into a couple of seeks, and the block cache keeps hot blocks in memory so most reads never reach HDFS - with BucketCache for when that cache is large enough that keeping it on the Java heap stops being viable.

Compaction is what keeps the file count bounded, and it exists because nothing else in the design can. Minor compactions fold small files into larger ones and cap read amplification. A major compaction rewrites every file in a region into one, and it is the only moment at which deleted cells, expired TTLs and versions beyond the retention limit actually stop occupying disk - a delete writes a tombstone that hides older cells rather than removing them, and until a rewrite passes over them they are all still there. The price is write amplification: every byte written is rewritten several times before it settles. That trade, and the strategies for scheduling it, are compaction.

Consistency: strong per row, absent across rows

HBase is strongly consistent per row, and single ownership is why that is cheap to provide. One server owns a region, every read and write for those rows passes through it, and so a read issued after an acknowledged write sees that write. There is no consistency level to choose per query, no read repair, no anti-entropy process, and no window in which two clients observe different values for the same row.

The guarantee stops at the row boundary, and stops hard. A Put touching twenty columns across three families is atomic - all of it applies or none of it does. checkAndPut and checkAndMutate provide compare-and-set on a single row, which is enough to build optimistic concurrency on, and Increment and Append are atomic read-modify-writes on one row. Across two rows there is nothing at all: no transaction, no rollback, no isolation, not even within a single table. When an invariant spans two entities the standard answer is to make them one row, and that is a large part of why HBase schemas denormalise as aggressively as they do.

Scans are the subtle case. A scanner fixes a read point when it opens and cells newer than that point are invisible to it, so each row comes back as a coherent snapshot of itself. The scan as a whole is not a snapshot of the table: it visits regions progressively and can perfectly well return rows that were written after it started. For an actual point-in-time image of a table, the mechanism is snapshots.

On the CAP axis this is a CP system and it is honest about it. When a RegionServer dies its regions are unavailable - not stale, unavailable - from the moment ZooKeeper expires the session until the log has been split and the regions reopened elsewhere, which is typically seconds to a couple of minutes. Nothing answers for those keys in the interim, and clients simply retry. Region replicas are the opt-in relaxation: secondary copies serve stale reads with timeline consistency while the primary is gone, trading the guarantee above for availability, on a per-table basis.

Where HBase sits next to Cassandra, Bigtable and a relational store

Against Cassandra. The dividing line is ownership versus quorum. Cassandra replicates each partition to several nodes, any of which can answer, and lets the caller choose per query how many must agree: availability under partition, tunable staleness, and a comparatively expensive read-modify-write because no single node is authoritative (lightweight transactions run Paxos to manufacture an authority). HBase puts one server in charge and gets strong single-row semantics and genuinely cheap atomic increments in return for an unavailability window when that server dies. Cassandra also carries no HDFS and no external coordinator, which makes it materially less work to operate. See HBase vs Cassandra and Cassandra consistency levels.

Against Bigtable. Same design - HBase was built from the Bigtable paper - with the operational surface removed: no HDFS to run, no ZooKeeper quorum, no masters, no JVM heap to tune, splitting and rebalancing handled by the service, capacity changed by moving a number. The data model and the key-design discipline transfer almost unchanged, which is why Cloud Bigtable is the usual landing spot for teams who want this model without the cluster. Compared head to head in HBase vs Bigtable; the other managed comparison is HBase vs DynamoDB.

Against a relational database. A relational engine offers many indexes, joins, a query planner and multi-row transactions, over a dataset one machine can hold. HBase offers one index and linear scale-out, and asks you to do the planner's job yourself at schema-design time by encoding the access path into the key. Phoenix layers SQL and secondary indexes on top, which is a real convenience and not a change of model: the indexes are additional tables it maintains on your behalf, with the write amplification and consistency caveats that implies.

When HBase is the wrong choice

The data fits on one machine. Below a few terabytes a single well-indexed PostgreSQL or MySQL instance beats an HBase cluster on latency, on query flexibility, and on engineer-hours by a margin that is not close. HBase starts earning its complexity in the high tens of terabytes; below that you are paying an operational tax for scale you are not using.

You need to query by more than one thing. There is one index. Every additional access path is a second table you maintain yourself, a Phoenix index, or a coprocessor - all of which mean a second write that can fail independently of the first, and therefore a reconciliation job you now own. If the query pattern is genuinely multi-dimensional, or not known in advance, that is a search engine or a relational database rather than a key you must guess correctly before the first row is written. Secondary indexes covers the options and what each one costs.

The workload is analytical. HBase stores whole rows, repeats the key on every cell, and answers one row at a time. Scanning a billion rows to compute an aggregate is something columnar files on object storage do an order of magnitude faster and cheaper. The mature architecture is usually both: HBase for the operational path, periodic export into a columnar table for analysis.

You need transactions across rows. Covered above, and worth restating as a filter rather than a detail: if the invariant cannot be folded into a single row, HBase has no answer for you.

You do not have the operations capacity. Production HBase means a ZooKeeper quorum, an HDFS cluster, redundant masters, and a fleet of JVMs whose garbage collector sits directly in your tail latency. Three nodes of HBase is very nearly all overhead. The managed equivalents exist precisely because this is the part teams underestimate.

The values are large. Cells of a few megabytes push the flush and compaction machinery hard, since every compaction rewrites the payload along with the key. MOB raises the practical ceiling to roughly the low tens of megabytes by storing those values off to the side. Above that, put the bytes in object storage and the pointer in HBase.

You need a hard sub-millisecond tail. Reads served from the block cache are fast, but GC pauses and compaction I/O put a floor under p99.9 that tuning moves rather than removes. If the SLA is stated in single-digit milliseconds at three nines, benchmark it before committing - and read the alternatives landscape before assuming HBase is still the default choice for the shape of problem you have.