HBase HFile Format — On-Disk Layout for Sorted Cells

Why it matters

HBase read latency is largely determined by HFile behavior. If the block cache hits, reads are microseconds. If they miss and go to HDFS, reads are milliseconds. Bloom filters can skip entire HFiles that provably do not contain the target row. HFile encoding and compression affect both storage cost and CPU cost per read.

Getting HFile configuration right — block size, compression, encoding, bloom filter type — is one of the most impactful HBase tuning choices.

The architecture

An HFile has four logical sections. The data section contains sorted key/value cells packed into blocks (default 64 KB uncompressed). The index section holds a hierarchical index that maps row keys to the block containing them, with root + intermediate + leaf indices for large files. The bloom filter section holds probabilistic membership filters for rows or row+column combinations. The metadata and trailer sections hold version info, block encoding options, and pointers to every other section.

The trailer is read first because it contains offsets and lengths of all other sections. From the trailer, the RegionServer can jump directly to the index, then to a specific data block, without reading the whole file.

How it works end to end

Point lookup flow: check the bloom filter first; if it says the row is absent, skip this HFile entirely. Otherwise consult the index to find which data block might contain the row, load that block (from block cache or HDFS), and scan within it. All this typically requires zero to two HDFS reads.

Scan flow: consult the index to find the first block containing the start row, then stream blocks sequentially until the end row is reached. Block prefetching hides HDFS latency: while the RegionServer processes one block, the next block is being fetched in the background.

Data block encoding (like DIFF or FAST_DIFF) compresses cell keys within a block by only storing differences from the previous key. This saves 40 to 70 percent of space for sorted keys with common prefixes. Compression (Snappy, ZSTD, GZ) further compresses each block. Encoding is fast; compression trades CPU for storage.

The four physical sections on disk

The logical picture above maps onto four physical regions laid down in a fixed order, because the writer streams the file out in a single append-only pass and can never go back. First comes the scanned block section: data blocks, leaf-level index blocks and bloom chunk blocks, interleaved in write order. These are exactly the blocks a full scan has to walk, which is why they share a region.

Second is the non-scanned block section: meta blocks and intermediate-level index blocks, which a sequential scan never touches. Third is the load-on-open section, and this one has an operational cost attached: everything in it is read into RegionServer memory the moment the store file is opened and stays resident for the life of the reader. That is the root data index, the meta index, the FileInfo map and the bloom filter metadata. Fourth and last is the fixed-size trailer.

The split is worth internalising because load-on-open memory is per-file and permanent. A RegionServer holding twenty thousand store files pays that cost twenty thousand times, whether or not a single read ever touches those files. It is one of the concrete reasons an unbounded HFile count hurts a cluster even when the block cache is behaving. Nothing in the format ever requires a reader to scan forward looking for structure: every region is located by an offset recorded somewhere else, and the whole chain of offsets terminates at the very end of the file.

Why the trailer is read first

Opening an HFile begins at the end of the file. The reader seeks to the last few bytes, pulls the format version out of the final word, and from the version it knows how large the trailer for that version is. It then reads the fixed-size trailer in full. Only after that does it know anything else about the file.

The trailer carries the offset of the load-on-open section, the offset of the FileInfo block, the number of entries in the root data index, the number of data index levels, the total cell count, the total uncompressed size, the compression codec, the class name of the key comparator, and the offsets of the first and last data blocks.

Every one of those is something the reader needs before it can interpret a single byte of the rest of the file, and none of it can be inferred from a data block. You cannot decompress a block without knowing the codec. You cannot binary search an index without knowing which comparator defines the key order. You cannot walk the index without knowing how many levels it has. So the format puts them all in one fixed-size record at a known position, and the position that is knowable without reading anything is the end.

This also produces a clean failure mode. A file truncated by a crashed write, or a zero-length file left behind by a failed flush, fails at open with a trailer or version error rather than silently returning partial data. The file is unreadable as a unit, which is the safe outcome; a format that started at byte zero would happily hand back the prefix that survived.

The block header and HBase's own checksums

Every block in the file, of every type, begins with the same header. In HFile v2 with HBase checksums enabled it is 33 bytes: an 8-byte magic naming the block type, a 4-byte on-disk size excluding the header, a 4-byte uncompressed size excluding the header, an 8-byte offset of the previous block of the same type, a 1-byte checksum type, a 4-byte bytes-per-checksum value, and a 4-byte on-disk data size including the header. Older v2 files written before HBase-managed checksums use a 24-byte header without the last three fields.

Two size fields are needed because the payload may be compressed: one tells the reader how many bytes to pull off disk, the other how large a buffer to allocate for the decompressed result. Since the index also stores each block's on-disk size, a single positioned read fetches header and payload together - the reader never has to read the header, then go back for the body.

Checksums that avoid reading the file twice

HDFS already checksums everything, but it stores those checksums in a separate file, so a verified block read is two IOs: the data and its checksum. HBase sidesteps that by computing its own CRC over each fixed-size chunk of the block (16 KB by default, per the bytes-per-checksum field) and writing those CRCs into the block itself. With hbase.regionserver.checksum.verify enabled, HBase opens the HDFS stream with checksum verification turned off and validates the block from data it already has in hand. One read instead of two, on the hottest path in the system. If a block fails HBase's own check, the reader falls back to reading it through HDFS checksum verification so a genuinely corrupt replica is still detected and reported rather than silently trusted.

How a seek resolves through the multi-level index

Start with the arithmetic, because it is what forces the index to have levels at all. An index entry is the first key of a block plus an 8-byte offset and a 4-byte length. With 60-byte row keys that is roughly 75 bytes per entry. A 10 GB HFile with 64 KB blocks holds about 160,000 blocks, so its complete index is on the order of 12 MB. Multiply by the store files on a RegionServer and a flat, fully resident index is obviously not viable.

So the index is built bottom-up and bounded. The writer accumulates index entries and, when a level's block would exceed hfile.index.block.max.size (128 KB by default), flushes it as a leaf index block into the scanned block section and promotes one entry to the level above. If the root level fits in that budget, the file has a single-level index: the root is the whole index and it lives in the load-on-open section. If not, an intermediate level appears. The trailer records how many levels resulted.

What a Get actually costs

A seek binary searches the in-memory root index to find which leaf or intermediate block could contain the key, reads that block (through the block cache, so a hot one is a memory hit), binary searches it, obtains the target data block's offset and on-disk size, reads that block, and finally seeks to the cell inside it. A single-level index means one block read for the data; a two-level index adds at most one more. Because leaf index blocks are cached exactly like data blocks, a working set that fits in cache turns the index walk into pure CPU.

The index is keyed on the first key of each block, not on every key. A lookup therefore resolves to the block whose first key is the greatest key not exceeding the target, and the reader confirms inside the block. This is why the index can be four orders of magnitude smaller than the data and still answer exactly.

Block size: index size against read amplification

BLOCKSIZE is a per-column-family property defaulting to 65536. It is a target, not a boundary: the writer finishes the cell in progress and then closes the block once it has passed the target, so blocks routinely overshoot and a single cell larger than the block size produces an oversized block on its own. A cell is never split across blocks.

Shrinking the block helps point reads. A Get transfers, checksums and decompresses one block to return one cell, so a 16 KB block does a quarter of the work a 64 KB block does for the same answer, and the cache holds the hot set at finer granularity. The bill arrives on the index side: halving the block size doubles the block count, doubles the index bytes, and pushes the file toward an extra index level and more permanent load-on-open memory.

Growing the block helps scans. Fewer, larger blocks mean fewer index entries, fewer positioned reads to stream a range, and a better compression ratio because the codec sees more data per window. The bill arrives on the point-read side: with 256 KB blocks, a Get for a 100-byte cell decompresses 256 KB to produce 100 bytes.

The practical split is by access pattern, per column family, not per table: latency-sensitive random-Get families go to 8-16 KB, scan-heavy analytical families go to 128-256 KB, and mixed workloads stay near the default. The block is also the unit the cache stores, admits and evicts, so block size sets cache granularity too - see HBase block cache architecture for what happens on the other side of that boundary.

Cell layout, and why the row key is repeated on every cell

An HFile contains no row records. It is a flat, sorted sequence of cells, each one independently comparable and independently seekable, ordered by row, then family, then qualifier, then timestamp descending, then type. That design decision is what makes the block index and the encodings possible, and it is also why storage overhead in HBase is so often underestimated.

KeyValue on disk
  4  key length
  4  value length
  |- key ------------------------------------------
  2    row length
  n    row bytes
  1    family length
  n    family bytes
  n    qualifier bytes      (length implied: keylen minus the rest)
  8    timestamp
  1    key type             (Put, Delete, DeleteColumn, DeleteFamily, ...)
  |- value ----------------------------------------
  n    value bytes
  |- tags (HFile v3 only) --------------------------
  n    length-prefixed tag list
                            (ACLs, visibility labels, cell TTL)

The fixed overhead is 20 bytes per cell before any of your data. On top of that, the row key, the family name and the qualifier are written out in full for every single cell. Take a 40-byte row key, a family named cf, a 12-byte qualifier and an 8-byte value: 20 + 40 + 2 + 12 + 8 = 82 bytes on disk to store 8 bytes of payload. A row with twenty qualifiers repeats that 40-byte row key twenty times.

This is where the folklore about HBase schema design comes from - single-character column family names, terse qualifiers, and the warning that wide rows are not free. Nothing about the format is wasteful by accident; the repetition is what buys independent addressability of every cell. The encodings below exist specifically to claw most of it back.

Data block encoding: prefix, diff, fast-diff, row-index

Data block encoding works inside a block, exploiting the fact that adjacent cells in a sorted file are nearly identical. It is set per column family via DATA_BLOCK_ENCODING.

PREFIX replaces each key with the length of the prefix it shares with the previous key plus the remaining suffix. In a wide row that erases nearly the whole repeated row key. DIFF adds to that: the column family is written once per block rather than per cell, the timestamp is stored as a delta from the previous cell's timestamp, and a flag byte lets common cases omit fields entirely. FAST_DIFF is DIFF plus value de-duplication - a flag marks a value byte-identical to its predecessor - and is tuned for decode speed rather than maximum ratio, which makes it the usual general-purpose choice. ROW_INDEX_V1 is the odd one out: it shrinks nothing, and instead appends an index of row offsets to each block so that seeking within the block is a binary search rather than a linear walk.

The asymmetry that makes encoding worth more than it looks

Encoded blocks stay encoded in the block cache. A block that is half the size on disk is also half the size in memory, so encoding multiplies effective cache capacity rather than just saving storage. Compression does not do this.

The cost is on seek. Under the delta encodings a cell's key is defined relative to its predecessor, so you cannot binary search inside a block - the reader decodes forward from the start of the block to reach the target cell. With 64 KB blocks that work is bounded, but it is real CPU on every Get, it scales with block size, and it is precisely the cost ROW_INDEX_V1 exists to remove for random-read families that can afford the bytes.

Compression versus encoding: two different layers

These are routinely conflated and they operate at different points in the pipeline. Encoding understands cell structure and rewrites keys. Compression treats the finished block as an opaque byte range and runs a generic codec over it: NONE, SNAPPY, LZ4, GZ or ZSTD, again per column family.

write:  cells -> data block encoder -> block bytes
              -> compression codec
              -> 33-byte header + compressed payload + CRC chunks -> HDFS

read:   HDFS -> header + payload -> verify CRC
              -> decompress
              -> block cache holds it here: uncompressed, still encoded
              -> decode cells on access

Note where the cache sits. Blocks are decompressed before they are cached, so compression buys disk footprint and HDFS read bandwidth but nothing in memory; encoding buys disk footprint and memory. That is the whole reason the two knobs are usually set together rather than treated as alternatives.

Codec choice follows the same latency-versus-ratio logic as everywhere else. SNAPPY and LZ4 give a modest ratio at very low CPU and are the safe default for anything latency-sensitive. ZSTD compresses substantially better for more CPU and suits cold or archival families. GZ compresses best and costs the most, which is defensible only for data that is written once and read rarely. NONE is the right answer only when the values are already compressed or encrypted, where a codec burns CPU to produce nothing.

The metadata a reader consults before it seeks

Several structures inside the file exist purely to let a reader decide not to do IO.

Bloom filter metadata - bit count, hash count and the chunk index - lives in the load-on-open section, so it is resident as soon as the file is opened. Structurally, the interesting part is that the bloom bits get no special treatment from the format: they are written into the scanned block section as ordinary blocks, carrying the same header as a data block, addressed by the same offset-plus-length machinery, verified by the same checksums and admitted to the same cache. A bloom probe is a block read like any other. A separate delete-family bloom is stored the same way and lets a reader skip a file holding no family-level delete markers. Sizing, false-positive rate and the ROW versus ROWCOL decision are covered in depth in HBase bloom filters.

FileInfo carries the rest. The file's time range and earliest-put timestamp let a scan bounded by time skip an entire file without touching the index at all - a cheap win that time-series schemas get for free. It also records the last key in the file, average key and value lengths, the maximum sequence id, the bloom type and encoding actually used, and whether the file was produced by a major compaction. Those last two matter when you are debugging: they describe what the file is, which is not necessarily what the current column-family configuration says.

v2, v3, and how compaction rewrites all of it

HFile v2 introduced the structure described in this article - block-level headers, the multi-level index, load-on-open, chunked blooms - replacing v1's single monolithic index that had to be read whole at open. v3 keeps that layout unchanged and adds per-cell tags: a length-prefixed tag list following the value, which is the carrier for cell-level ACLs, visibility labels and cell TTLs, along with support for HFile-level encryption. It is selected by hfile.format.version and is the default on modern HBase.

The critical consequence of immutability is that every property discussed here is frozen at write time from the column-family configuration in force when that flush or compaction ran. Changing BLOCKSIZE, DATA_BLOCK_ENCODING, COMPRESSION or BLOOMFILTER changes nothing about existing files. New files pick up the new setting; old files keep theirs until something rewrites them, which is why a store can contain files with three different encodings at once and why a major compaction is the only operation that guarantees the whole store converges on the new schema.

Compaction is exactly that rewrite: a merging scanner reads the input files in key order and feeds a fresh writer, so the index, the blooms, the encoding, the compression and the checksums are all rebuilt from scratch rather than copied. A region split, by contrast, rewrites nothing at first - it creates reference files pointing at the top or bottom half of the parent's HFile, and the physical split only happens when the daughter regions next compact. Compaction policy, scheduling and the amplification tradeoffs behind it are covered in HBase compaction.

Setting it, and reading a real file

Everything that shapes the on-disk layout is a column-family attribute:

create 'events', \
  {NAME => 'e', BLOCKSIZE => '16384', COMPRESSION => 'SNAPPY', \
   DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER => 'ROWCOL'}

# applies to files written from now on, not to existing ones
alter 'events', {NAME => 'e', DATA_BLOCK_ENCODING => 'FAST_DIFF'}
major_compact 'events'    # forces every existing file to be rewritten

To see what a file actually contains rather than what you configured, use the HFile pretty printer. It reads the trailer and load-on-open section and prints them:

hbase hfile --printmeta --file /hbase/data/default/events/<region>/e/<file>
hbase hfile --stats    --file /hbase/data/default/events/<region>/e/<file>

# run `hbase hfile` with no arguments for the full option list on your version

The metadata dump is where you confirm what the file really is: format version, compression codec, data block encoding, bloom type and parameters, index depth, cell count and time range. That is how you verify an alter plus compaction actually took effect, rather than trusting the schema. The statistics dump reports key and value length distributions, which is often the fastest route to discovering that a table's real problem is 200-byte row keys carrying 12-byte values rather than anything you can fix with tuning.