Why architecture matters here
HBase architecture matters because HBase is unusually sensitive to operational choices. Row key design, region size, compaction strategy, and MemStore tuning all have order-of-magnitude effects on throughput and latency. A well-designed HBase cluster serves millions of ops per second; a badly designed one throttles at ten thousand.
Cost matters more than most NoSQL stores. HBase's dependence on HDFS means you pay for triple-replicated storage; row-level operations are inexpensive but wide scans and inefficient row keys chew through I/O quickly. Understanding compaction is critical: a compaction backlog wastes IOPS and eventually destabilizes the cluster.
Reliability is where HBase shines when it is tuned and disappoints when it is not. RegionServers that go down bring regions offline for the recovery period; well-tuned clusters recover in seconds, poorly tuned ones stall. Region assignment mechanics, master failover, and WAL replay behavior all determine the actual availability.
The architecture: every piece explained
Walk the diagram from top to bottom.
Client. HBase clients are thick — they cache the region-to-RegionServer map, contact RegionServers directly, and handle retries. On the first request the client asks ZooKeeper for the location of the meta table, then looks up the region for the target row, then contacts the RegionServer. Subsequent requests hit the cache; misses trigger a re-lookup.
ZooKeeper. The coordination service. It stores the master's location, the meta table's location, and health information about RegionServers. HMaster and RegionServers each maintain sessions with ZooKeeper; failure to renew a session is how the cluster detects node death.
HMaster. The coordinator. It handles region assignment, splits, and cluster-wide operations like table creation. HMaster is not on the read/write hot path — a running cluster can survive a brief HMaster outage. But you need HMaster for assignment, so long outages block recovery from other failures.
hbase:meta table. A special HBase table that maps region names to RegionServers. Clients read it to find where a row lives. It is itself a region, hosted on one RegionServer; its location is stored in ZooKeeper. This design lets HBase scale to billions of regions.
Region Assignment. HMaster's AssignmentManager decides which RegionServer hosts which region. Rebalancing moves regions to even out load. Region-server failures trigger re-assignment: the master detects the failure via ZooKeeper session loss, replays the failed server's WAL to recover any writes not yet flushed to HDFS, and assigns the regions to surviving servers.
RegionServers. The workers. Each RegionServer hosts many regions (typically 20-100 per server). It receives read and write requests for its regions. Writes go to the WAL first, then the MemStore. Reads consult the BlockCache, MemStore, and HFiles.
WAL (Write-Ahead Log). Every write is appended to a per-RegionServer WAL on HDFS before the client is acknowledged. If a RegionServer crashes, the master replays the WAL entries for each affected region to recover unflushed writes. WAL is durable but adds latency; you can trade durability for speed by disabling WAL for specific writes (not recommended).
MemStore. An in-memory sorted map per region per column family. Writes go into the MemStore after WAL append. When the MemStore fills up (default 128 MB), it flushes to an HFile on HDFS.
HFiles. Immutable, sorted, indexed files stored on HDFS. Each region accumulates many HFiles over time from repeated MemStore flushes. Reads must merge across HFiles plus the MemStore; compaction reduces the count over time.
Compaction. Minor compactions merge a few small HFiles into a larger one; major compactions merge all HFiles for a region into one, purging deleted rows and expired versions. Compaction is essential for read performance but competes with reads and writes for I/O.
Region Splits. When a region grows past a threshold (default ~10 GB), HBase splits it into two. The split is fast (metadata operation) but the follow-on compactions can be I/O heavy. Pre-splitting on table creation avoids the "hot spot" problem where a single region receives all initial writes.
BlockCache and BloomFilter. The BlockCache holds recently-read HFile blocks in memory. Bloom filters let reads skip HFiles that definitely don't contain the target row. Both are essential for hot-key read performance.
End-to-end read and write flow
Trace a write. A client writes row key "user-42" to table "profile". The client's cached meta says the region containing "user-42" is on RegionServer B.
The client sends the Put to RegionServer B. RegionServer B appends the write to its WAL and syncs to HDFS (durability). It inserts the write into the MemStore of the target region. It acknowledges to the client.
The MemStore fills over time. When it crosses the flush threshold, RegionServer B writes an HFile to HDFS containing the sorted MemStore contents. The MemStore is emptied. The region now has one more HFile.
Trace a read. The client asks for "user-42". Cached routing sends the Get to RegionServer B. B checks its BlockCache; if the block isn't cached, it consults bloom filters on the region's HFiles. Non-matching HFiles are skipped. Potentially-matching HFiles are read and merged with the MemStore contents to find the freshest value. The result flows back to the client.
Region grows. When the region hits 10 GB, HBase splits it into two child regions. The split is a metadata operation; existing HFiles are shared. Follow-on compaction gradually rewrites the files to be split-aligned.
RegionServer B fails. ZooKeeper session expires. HMaster detects the failure, finds surviving RegionServers with capacity, and assigns B's regions to them. Before serving reads or writes for each region, the new host replays the corresponding WAL entries. Recovery time is proportional to WAL size and target replay parallelism.