Why it matters
Block cache configuration is one of the largest levers for HBase read latency. A read-heavy workload with a small block cache sees frequent HDFS reads and slow response. A properly sized cache pins the working set in memory and keeps p99 latency low.
Off-heap BucketCache also unlocks HBase deployments on modern high-memory hardware. Without it, cache size is bounded by JVM heap, which is bounded by GC-tolerable sizes (32 GB or so). With BucketCache, you can cache hundreds of GB per host.
The architecture
LRU BlockCache is the default on-heap cache. It stores decoded HFile blocks in an LRU-eviction data structure allocated from the JVM heap. Simple and effective, but heap-bound: larger caches produce longer GC pauses.
BucketCache is an off-heap alternative. Blocks are serialized into fixed-size buckets outside the JVM heap. This unlocks arbitrarily large caches without GC impact. Cost is a small CPU overhead for serialization on each access.
How it works end to end
On read, the RegionServer computes the HFile block that contains the target row key from the HFile index. It first checks the block cache. If present (cache hit), the block is returned immediately and the scan proceeds. If absent (cache miss), the RegionServer reads the block from HDFS, stores it in the cache, and then uses it.
Combined mode uses LRU BlockCache for the small, hot working set and BucketCache for the larger, less-hot cache. This gives fastest access to the very hottest blocks (in on-heap) while extending cache size with off-heap.
Cache-on-write puts every newly-flushed block into cache automatically. This is useful for tables with strong temporal locality where new data is queried right away.