Why architecture matters here
The architecture matters because the gap between a cache hit and a cache miss is the gap between HBase being a low-latency store and being a slow one. A hit is a memory lookup measured in microseconds; a miss is an HDFS read that may involve a network hop to a DataNode, a disk seek, and block decompression, measured in milliseconds — three or more orders of magnitude worse, and with far worse tail behavior under load. For any read-serving workload, the hit ratio is the single most important performance number, because it directly sets what fraction of reads pay memory cost versus disk cost. The block cache exists to push that ratio as high as the working set and the available memory allow, and everything about its design is in service of protecting hits for the data that is actually hot.
It matters because caching at block granularity turns spatial locality into performance for free. HBase stores rows sorted by key, so the cells of one row and the rows near it in key order sit together in the same or adjacent HFile blocks. When a read pulls a block into the cache, it warms not just the requested cell but every neighbor in that 64KB block — which is precisely what the next point read of a nearby key, or a short range scan, will ask for. This is why block-level caching is so effective for HBase's sorted, wide-row data model: one miss populates the cache for a whole neighborhood of likely-next reads, so locality of access converts directly into a rising hit ratio.
The architecture also matters because not all cached blocks are equally valuable, and a naive single-LRU cache would let the wrong ones win. A giant one-time scan reads millions of blocks exactly once; under plain LRU those recently-touched-but-worthless blocks would evict the genuinely hot data that thousands of point reads depend on, tanking the hit ratio for everyone. HBase's three-priority scheme is the defense: single-access blocks (like scan blocks) occupy a lower-priority tier and are evicted first, multi-access blocks earn promotion by being read again, and in-memory blocks are protected. This is what stops a batch scan from destroying interactive read latency, and it is the reason the cache is priority-aware rather than a flat LRU.
Finally, it matters because index and bloom blocks are a different category of value entirely, and treating them like data blocks wastes the biggest win. Every HFile carries a block index (to find the right data block for a key) and, usually, bloom filters (to skip HFiles that cannot contain a key). These are small, are consulted on every read of that HFile, and being in cache turns a would-be extra disk seek into a memory lookup on the read path's critical section. Caching index and bloom blocks is almost always worth it regardless of the cache pressure on data blocks, because their hit avoids I/O on essentially every request — which is why the architecture prioritizes them and the operational advice is to keep them cached even when data caching is tight.
The architecture: every piece explained
Top row: the read-path lookup and populate cycle. A read request — a get or a scan — needs the cells in some HFile block. HBase performs a block cache lookup keyed by the HFile and block offset. On a hit, the decoded block is already in memory and the cells are served with no disk I/O — the fast path that a read-serving workload lives on. On a miss, HBase reads the block from the HFile on HDFS, decompresses it, serves the cells, and then caches the block so subsequent reads of any cell in it become hits. This read-and-populate cycle is what warms the cache from live traffic: the cache contents track the actual working set rather than being pre-loaded.
Middle row: priorities, eviction, and placement. Cached blocks are held under three priority levels. A block read for the first time enters as single-access; if it is read again it is promoted to multi-access, marking it as more valuable; blocks from a column family flagged IN_MEMORY enter the protected in-memory tier. Within each tier, LRU eviction reclaims the least-recently-used blocks when the cache is full, and the tiering ensures single-access blocks are sacrificed before multi-access before in-memory. The cache itself can sit on-heap — the classic LRU block cache, simple but adding garbage-collection pressure as it grows — or off-heap in a bucket cache, which sidesteps GC by managing its own memory at the cost of extra copying on access. The on-heap/off-heap choice is the central sizing decision at large cache footprints.
Bottom-left: what gets cached and why. HFiles contain data blocks, index blocks, and bloom blocks. Data blocks hold cells; index blocks locate the right data block for a key; bloom blocks let a read skip an HFile that cannot contain the key. Index and bloom blocks are small and consulted on essentially every read, so caching them is almost always worthwhile — their presence in cache removes a disk seek from the critical path of nearly every request, a far higher return than caching any individual data block.
Bottom-right and ops: the pollution risk and the levers. A large scan reads many blocks once, and if those blocks are cached they cause scan cache pollution — evicting hot single- and multi-access data that interactive reads depend on. The remedy is to run such scans with block caching disabled so they do not populate the cache. The ops strip names the tuning surface: sizing the block cache against the memstore (both compete for region-server memory, and over-allocating one starves the other), setting the IN_MEMORY flag on small, hot column families, passing cacheBlocks=false on bulk scans, and monitoring the hit ratio as the primary health metric of the read path.
End-to-end flow
Trace reads against a region server hosting a user-profile table — hot point reads, a cold analytical scan, and a small pinned lookup table — and watch the cache protect what matters.
Cold start warms the cache: the region server restarts with an empty block cache. The first wave of profile get requests all miss: each reads its block from the HFile on HDFS, decompresses it, serves the cell, and inserts the block as single-access. Latency is high during this warmup because every read pays disk cost. But because rows are sorted by user key, each miss warms a 64KB neighborhood, so nearby subsequent reads increasingly hit. Within minutes the hot working set of active profiles is resident and the hit ratio climbs from near zero toward its steady-state high value.
Promotion protects the truly hot: some profiles are read far more than others — celebrities, popular accounts. The second read of such a block promotes it from single-access to multi-access, marking it more valuable than blocks touched only once. As the cache fills and eviction begins, LRU reclaims single-access blocks first, so the repeatedly-read hot blocks survive while the long tail of one-off reads churns through the lower tier. The priority structure is quietly sorting the genuinely hot data above the incidental.
The scan threatens pollution: an analytics job launches a full-table scan to export every profile. Read naively, it would pull millions of blocks into the cache as single-access, and even though they sit in the lowest tier, the sheer volume would evict the hot multi-access working set and crater interactive latency. Because the job is configured with cacheBlocks=false, its blocks are read and served but not inserted into the cache — the scan streams through without disturbing the hot set. The point reads' hit ratio stays high right through the scan, which is exactly the isolation the cacheBlocks flag exists to provide.
The pinned lookup table: a small reference table — say country and currency codes — is read constantly by every request but is tiny. Its column family is flagged IN_MEMORY, so its blocks enter the protected in-memory tier and are evicted only after everything else. This guarantees that the ubiquitous lookups are always hits regardless of the churn in the data tiers, spending a small, bounded amount of cache to eliminate misses on a table that would otherwise be re-read from disk under memory pressure.
Index and bloom stay resident: underneath all of this, the HFiles' index and bloom blocks are cached and stay cached, because they are consulted on nearly every read. When a get for a rarely-accessed profile misses the data cache, the bloom block still answers 'this HFile might contain the key' from memory and the index block still locates the data block from memory, so the read pays only one data-block disk seek instead of several. The cache's biggest, most consistent win is not any single hot row but the index and bloom blocks that shave I/O off the critical path of every request the region server serves.