Parquet Format — Cross-Engine Columnar Standard

Why it matters

Parquet's portability is its biggest value. A table in Parquet can be read by every major analytical engine without conversion. This is what enables data lakes with multiple compute engines against the same storage.

The architecture

A Parquet file is divided into row groups (typically 128 MB). Each row group contains column chunks; each column chunk contains pages (data page, dictionary page, index page). Data pages are encoded per-column with schemes chosen at write time.

File footer holds metadata: row group locations, column statistics, page statistics.

How it works end to end

Column pruning is automatic like ORC. Predicate pushdown uses row group stats (min/max per column) to skip entire row groups.

Dictionary encoding is the biggest single win for low-cardinality columns. A country column with 200 unique values becomes tiny.

Page indexes let readers skip within a column chunk based on column min/max per page. Newer readers use this for fine-grained skipping.

The three-level hierarchy and the job each level does

A Parquet file is not simply "columns on disk". The rows are cut horizontally first, then vertically, then cut again. Each of the three levels exists to solve a different problem, and confusing them is the source of most bad tuning decisions.

Row group - a horizontal slice holding every column for some number of rows. Cutting horizontally first is what makes a file splittable: one task can take one row group and have every column it needs inside a single contiguous region, with no coordination against another part of the file. It also puts a ceiling on writer memory. A writer cannot emit a column chunk's offset until it knows the encoded size of everything before it, so it buffers a whole row group and flushes at the boundary. Without row groups a writer would need the entire file resident in heap.

Column chunk - within one row group, all values of one column, stored contiguously. This is the unit of column projection. Reading 3 columns of 80 means issuing 3 byte ranges per row group and never touching the other 77 chunks. On object storage that maps one-to-one onto ranged GETs, which is why projection is nearly free there rather than merely cheap.

Page - within a column chunk, a run of values, traditionally around 1 MB uncompressed in parquet-mr's defaults (verify against whatever writes your files). The page is the unit of encoding and of compression, and every page is independently decodable. That independence is what makes fine-grained skipping possible at all: you cannot seek into the middle of a compressed byte stream, but you can skip an entire page without decompressing it.

The nesting is strict. Pages never span column chunks, column chunks never span row groups, and no value is stored twice. That is precisely what allows the file's entire index to be nothing but a table of offsets and lengths.

What the footer holds, and why wide schemas hurt

Parquet files open and close with the four-byte magic PAR1. Immediately before the trailing magic sits a four-byte little-endian footer length, and before that the Thrift-serialized FileMetaData. The open sequence is therefore fixed: read the last 8 bytes, learn the footer length, read that many bytes back from the end, parse. There is no header to consult, because the writer did not know any of this until it finished.

The footer carries the schema flattened depth-first into leaf columns (only leaves have data), the file's row count, the writer identity string in created_by, an arbitrary key/value map where engines stash their own schema JSON or field IDs, and then - the bulk of it - for every row group, its byte size and row count, and for every column chunk inside it: the offset of the first data page, the offset of the dictionary page if there is one, compressed and uncompressed sizes, the codec, the list of encodings actually used, and the statistics.

That last part is O(row groups x columns), and it is a real operational limit. A 30-column table with 8 row groups has a footer nobody notices. A 4,000-column feature table with 200 row groups carries metadata for 800,000 column chunks, and every task that opens the file deserializes all of it - including the 3,995 columns the query never projects. Wide-schema Parquet tables can spend a startling share of query time inside Thrift deserialization. The fix is fewer and wider row groups, and splitting absurd schemas across files; no reader optimization rescues a footer that large.

The footer's position also sets the floor on object storage: at least two round trips per file before a single value is read, often three once the page index is fetched. A directory of small files pays that toll per file, which is the mechanical reason small files punish Parquet harder than they punish plain text.

Column chunk statistics - what the footer can prove

Each column chunk records min_value, max_value, null_count, and optionally distinct_count. In Parquet these live in the footer alongside the offsets, not in a separate index region inside the data area - which is why a reader gets every row group's statistics for every column in the same read that told it where the data is.

null_count is the most underrated of the four. A count of zero lets a reader drop the row group for IS NULL outright; a count equal to the row count answers IS NULL without reading a byte and drops the group for IS NOT NULL.

Ordering is not automatic. A minimum and a maximum only mean something if writer and reader agree on how that type is ordered, and that agreement was not always there: unsigned integers, DECIMAL, and UTF-8 byte arrays were compared as signed bytes by some early writers, producing ranges that are simply wrong. Parquet's answer was to deprecate the original min/max fields in favour of min_value/max_value with an explicitly declared column order, and to have readers consult created_by before trusting legacy statistics. A reader that cannot establish the ordering for a type must ignore the statistics and read the data. That is correct, and it is indistinguishable from the outside from "pushdown is broken".

Truncation. Writers may truncate long byte-array minima and maxima to keep the footer bounded. A truncated bound is a prefix, so the recorded range is always wider than the true range, never narrower - skipping stays correct and merely becomes less selective. Equality against a long string key gets essentially nothing from it. distinct_count, meanwhile, is usually absent; do not design around it.

The granularity problem. A Parquet row group is commonly 128 MB - millions of rows. Compare that with ORC's fixed 10,000-row index groups. Parquet's row-group statistics therefore need far stronger physical clustering before they exclude anything: a value that occurs once per million rows lands inside every row group's range, and the min/max test passes everywhere. This coarseness is the specific problem the page index was invented to solve.

Page indexes - the skip granularity that arrived later

Every data page header has always carried its own statistics, and they have always been useless for skipping. Page headers are interleaved with page data, so reaching the header of page 40 means walking pages 1 through 39. Statistics you can only obtain by reading the bytes you were trying to avoid are not an index.

The fix was to hoist them out of the data area into two structures written near the footer, one pair per column chunk:

ColumnIndex - parallel arrays across the chunk's pages: minimum values, maximum values, a null_pages boolean per page, null counts, and a boundary_order flag declaring whether the pages' ranges run ascending, descending, or unordered.

OffsetIndex - per page, its file offset, its compressed length, and first_row_index, the ordinal of the first row it contains.

ColumnIndex decides which pages can possibly match; OffsetIndex says where those pages are and which rows they start at. Together they let a reader turn a predicate into a set of byte ranges plus a row mask, fetch only those ranges, and then read only the corresponding row ranges of the projected columns. When several columns are filtered, the reader intersects their surviving row ranges first. This is what keeps Parquet competitive on selective queries in spite of 128 MB row groups.

boundary_order is the part that gets overlooked. When it declares ascending order the reader can binary-search the page ranges rather than scanning them, which matters once a column chunk holds thousands of pages - exactly the case where the index was most needed.

Two conditions have to hold, and neither is guaranteed. The writer must have emitted the indexes: older files and some non-Java writers have none. And the reader must be configured to consult them; several engines gate this behind a flag. There is no way to infer either from the query plan - inspect the file.

Bloom filters, and when the dictionary already did the job

Minima and maxima cannot answer WHERE user_id = 'a3f9c...' on an unsorted high-cardinality column, because the value sits inside every row group's range. Parquet's answer is an optional bloom filter per column chunk, written after the data with its offset recorded in the chunk metadata so a reader can fetch the filter without touching a single page.

The construction is a split-block bloom filter: the filter is an array of 32-byte blocks, each holding eight 32-bit words, and a hash selects one block in which exactly one bit per word is set. Every bit of a probe therefore lands in one cache line, so a lookup is one memory access rather than eight scattered ones. It is a slightly worse filter mathematically and a much faster one in practice.

The part worth internalizing is when a bloom filter is redundant. If a column chunk is fully dictionary-encoded, the dictionary page is already an exact membership test - it is small, it sits at a known offset, and it has no false positives at all. A bloom filter over that chunk adds nothing. Bloom filters earn their bytes precisely on the chunks where dictionary encoding fell back to plain, which is to say on high-cardinality chunks: the same chunks where the writer decided the dictionary had grown too large.

So the sensible configuration is narrow and deliberate: enable filters on the specific columns you do point lookups on, size them against the expected distinct count per row group rather than per file, and expect them to be quietly ignored on the row groups where the dictionary survived. Sizing them against the whole table's cardinality is the common mistake - it inflates every filter in the file for no gain.

Encodings per physical type, and what the codec ever sees

Parquet has very few physical types - booleans, 32- and 64-bit integers, float, double, byte arrays, fixed-length byte arrays, plus the legacy INT96. Everything richer (DATE, DECIMAL, TIMESTAMP, STRING, UUID, JSON) is a logical type annotation layered on one of them. Encodings are chosen against the physical type, per column chunk, at write time.

PLAIN writes values as they are; it is the fallback everything degrades into. RLE_DICTIONARY gives the chunk a dictionary page of distinct values and stores bit-packed indices into it, at ceil(log2(dictionary_size)) bits per value with run-length escapes for repeats - a country column with 200 distinct values costs one byte per row before the codec has done anything. DELTA_BINARY_PACKED stores differences between consecutive integers in mini-blocks, each with its own bit width, which collapses timestamps, monotonic identifiers and counters to a couple of bits per value. DELTA_BYTE_ARRAY stores a shared-prefix length plus a suffix against the previous value, which is why sorted URLs, file paths and hierarchical keys shrink dramatically. DELTA_LENGTH_BYTE_ARRAY splits the length array away from the concatenated bytes so the lengths can be delta-packed on their own. BYTE_STREAM_SPLIT transposes floating-point data so that all first bytes sit together, all second bytes together, and so on; floats have no run structure, but their exponent bytes are highly repetitive once separated, and this is what gives a general codec anything at all to work with on a column of doubles.

Dictionary fallback is the behaviour to know by heart. The writer accumulates a dictionary as it fills a chunk, and when that dictionary exceeds a size limit (1 MB was the long-standing parquet-mr default; confirm yours) it gives up: pages already emitted stay dictionary-encoded, and everything after is written PLAIN. The decision is per column chunk, so one column can be dictionary-encoded in row group 3 and plain in row group 4 of the same file. Two consequences follow. File size becomes sensitive to row group size for any column near the threshold, and the dictionary-as-exact-filter shortcut vanishes on exactly the chunks where a point lookup most needed it.

Only then does the codec run - SNAPPY, ZSTD, GZIP, LZ4, BROTLI - and it runs on already-encoded page bytes. It never sees your values. It sees bit-packed dictionary indices and delta mini-blocks, from which the type-aware redundancy has already been removed. This is why the codec is the smaller decision: swapping SNAPPY for ZSTD moves file size by a fraction, while losing dictionary encoding on a wide string column moves it by a multiple. Treat codec choice as a CPU-against-bytes trade - heavier codecs win when you are billed per byte scanned or are network-bound, and lose when the data is on local NVMe and the scan is already CPU-saturated.

Repetition and definition levels

The three-level container is the part everyone draws. The levels are the part almost nobody explains, and they are the entire reason Parquet stores nested and optional data without flattening it into repeated copies of the parent.

Only leaves of the schema tree get column chunks. In the schema below, user.phones.label is a column; user and user.phones are not stored at all. So a reader is handed a flat run of label strings and must rebuild records from it, with no record boundaries written anywhere. Two small integers per value carry every bit of structure needed to do that.

message Doc {
  required int64  id;
  optional group  user {
    repeated group phones {
      required binary number (STRING);
      optional binary label  (STRING);
    }
  }
}

Working out the maximum levels

Walk the path from root to leaf and count. The maximum definition level gains 1 for every optional or repeated node on the path; the maximum repetition level gains 1 for every repeated node.

What a definition level actually means

For label, whose maximum is 3, the definition level says exactly how far down the path existence got before it stopped:

Only a level equal to the maximum stores a value. That is the whole null encoding: a null costs its level integer and nothing more. And it is how three genuinely different kinds of absence - missing parent, empty list, null leaf - stay distinguishable without a single extra column or sentinel value.

What a repetition level means

The repetition level answers a different question: at which repeated ancestor does this value continue a list? A level of 0 always means "this value begins a new top-level record". For phones, whose maximum is 1, a repetition level of 1 means "another phone belonging to the record already in progress". Nest a repeated group inside another - phone numbers inside addresses - and level 2 would mean another element of the inner list, level 1 a new inner list within the same outer element, level 0 a new record. Record reconstruction is a small state machine driven purely by these integers.

What the machinery buys, and what it costs

Levels are written per page, ahead of the values, RLE and bit-packed at ceil(log2(max_level + 1)) bits. For a flat required column both maxima are 0, the width is zero bits, and the levels cost literally nothing - which is exactly why this mechanism is invisible in ordinary flat tables. For label above, definition levels take 2 bits and repetition levels 1 bit, and a long run of records with no phones at all collapses into a single RLE run.

What you gain: reading user.phones.number reads that one column chunk and its levels. No parent structure is materialized, no reconstruction of siblings occurs, and no join is performed. The cost is proportional to the leaf you asked for, not to the depth or width of the struct containing it - one field out of a 40-field struct costs one field.

What you pay: statistics and page indexes exist per leaf column chunk, but a row group holds whole records, and a predicate over a repeated field ("some phone has label = 'work'") is an existential over a list rather than a comparison against a scalar. Engines vary widely in whether they push such predicates down at all, and several simply do not. Deeply nested columns also fall off the fast path in some vectorized readers. If a nested field is a hot filter, promoting it to a top-level column is a legitimate physical-design decision, not a hack.

Schema evolution and what is safely additive

The format itself promises almost nothing here. Parquet records a schema per file; what happens when a file's schema disagrees with the table's is a reader decision and differs by engine. The rules worth trusting:

Adding a column is safe. Old files simply lack the chunk and readers materialize nulls. This depends on resolution being by name, or by field ID where a table format supplies them.

Dropping a column is safe for any reader that projects by name - the chunk is never requested, and no rewrite is needed.

Reordering is safe under name-based resolution and catastrophic under position-based resolution. Hive's Parquet handling has historically resolved by position under some table configurations, which turns a harmless-looking column reorder into silent data corruption: columns read as each other, no error raised. Iceberg and Delta sidestep this entirely by writing and matching stable field IDs.

Renaming is not evolution. Without field IDs it is a drop plus an add, and the old data is unreachable under the new name.

Widening types is a reader capability, not a format guarantee. INT32 to INT64 and FLOAT to DOUBLE are supported by some engines and not others. Test on your engine; do not assume.

Required to optional is safe going forward; optional to required is not. This is where the levels section pays off. Making a column non-nullable reduces its maximum definition level, which changes the bit width and the meaning of every level value already written. Old files then encode something the new schema cannot express. A nullability change is a physical change to how every value in that column is stored, not a metadata edit - and that is the single most common evolution mistake in Parquet tables.

Row group sizing against block size and writer memory

The historic guidance was one row group per HDFS block, traditionally 128 MB, so that a task handed one block reads exactly one complete row group from local disk with no remote block fetch. On HDFS that instinct is still right. On object storage there are no blocks at all; what matters is the count and size of ranged GETs, and larger row groups mean fewer, larger reads - which is what object stores reward.

Pulling the other way is writer memory, and this is where jobs actually die. The writer must buffer an entire row group - every column's encoded pages plus every column's dictionary - before it can flush, because a column chunk's offset is unknowable until everything preceding it is final. Peak writer heap is therefore roughly the row group target multiplied by the number of files open at once. A dynamic-partition insert with 200 partitions live in one task and a 128 MB target needs vastly more heap than the container has, and the symptom is an out-of-memory error inside the writer that looks nothing like a sizing problem. The two real fixes are lowering the row group target for wide or many-writer jobs, and sorting or distributing by the partition column so that far fewer writers are open simultaneously - see Hive dynamic partitioning.

Small files are the same constraint viewed from the other end. A 4 MB Parquet file is one small row group: its statistics cover too few rows to exclude anything, its footer round trips cost as much as reading the data, and its dictionary never amortizes. Compacting to files of a few hundred megabytes recovers all three at once - see the small-file problem.

What the Parquet reader can and cannot evaluate

How a WHERE clause is split into pushable and residual parts, and compiled into a SearchArgument handed to the storage layer, is covered in Hive predicate pushdown. What belongs here is the Parquet-specific list of things that quietly produce no skipping whatsoever:

A cast wrapped around the column. WHERE CAST(id AS STRING) = '42' compares against something the footer does not hold. Casting the literal side is fine; casting the column side destroys the comparison.

Pattern matching. LIKE 'abc%' is in principle a prefix range that min/max could answer, but many readers never implement it. LIKE '%abc' never can.

Legacy INT96 timestamps. They carry no logical type and no defined ordering, so their statistics are typically discarded outright. Tables written by older Impala or older Spark configurations are full of them; rewriting to INT64 timestamps is what restores skipping.

Truncated string bounds. Because truncation widens the range, an equality test against a long key is inconclusive and the reader is obliged to read.

Columns absent from older files. The predicate resolves against null, which most engines treat as unknown rather than false - so the file is not skipped.

Untrusted writers. If created_by identifies a writer whose statistics the reader does not trust for that type, the statistics are ignored wholesale.

In every one of these cases the symptom is identical and unhelpful: the plan still shows a filter and the scan still reads everything. The diagnostic is not the plan but the ratio of rows read to rows returned in the query profile. A ratio near one means nothing was excluded, whatever the plan claims.

Parquet against ORC, on the axes that actually differ

Most of what gets compared is shared. Both formats put column-oriented data inside horizontal groups, keep min/max and null statistics at more than one level, support optional bloom filters, apply dictionary and run-length encoding before a general codec, do column projection, and feed vectorized readers. Benchmark differences of a few percent are usually measuring writer configuration, not format. The differences that survive scrutiny are these.

Default skip granularity. ORC's row index has recorded 10,000-row groups with stream positions since the beginning, so seeking to a sub-stripe boundary is native and always available - see the ORC format article for how those positions work. Parquet's counterpart is the page index: newer, optional, and keyed to pages sized in bytes rather than a fixed row count. Between a writer and reader that both support it the gap largely closes; on older files it does not exist at all.

Nesting model. Parquet uses repetition and definition levels; ORC gives every node in the type tree its own PRESENT stream. ORC's is easier to implement and to reason about. Parquet's is markedly more compact for sparse nested data, because an absent subtree costs one small integer instead of a bit in every descendant's present stream.

Transactional integration in Hive. Hive ACID is built on ORC's embedded row identity, so if you need UPDATE, DELETE or MERGE inside Hive itself the format decision is already made. Iceberg changes the calculus, because it implements row-level deletes above the file format and lets Parquet participate on equal terms.

Ecosystem reach. This is the real one, and it is what the title of this article means. Parquet is what Spark, Impala, Trino, DuckDB, Arrow-based Python tooling and every cloud warehouse's external-table reader handle first and best. ORC support outside the Hive, Tez and Spark orbit is thinner and tends to lag. If a second engine, or any non-JVM tool, will ever read these files, that consideration outweighs every layout detail above.

A workable rule: ORC for Hive-native ACID tables and Hive-only warehouses; Parquet for anything a second engine will touch - which in practice describes most new tables.

Reading a real file before you trust it

Every claim above is checkable against an actual file with the Parquet CLI, and doing so takes under a minute.

# footer: schema, row groups, per-chunk sizes, encodings, codec, statistics
parquet meta part-00000.zstd.parquet

# page-level min/max and offsets; empty output means no page index was written
parquet column-index part-00000.zstd.parquet

# page geometry inside each column chunk
parquet pages part-00000.zstd.parquet

Four things are worth looking at every time. Row group count and size - a single row group in a 4 MB file is the small-file problem in miniature; hundreds of tiny row groups mean the writer's target was overridden by something upstream. The encodings list per column chunk - a string column you expected to be dictionary-encoded showing only PLAIN means the dictionary hit its limit, and you have lost both file size and the exact-membership shortcut for point lookups. Presence of ColumnIndex offsets - absent means no page-level skipping is possible no matter how the reader is configured. Statistics on your filter column - if every row group's min and max are identical to the column's global range, the data is not clustered on that column and row-group skipping will never fire, whatever the query plan implies.

The reason this loop matters is that Parquet gives no feedback at write time. A misconfigured writer produces a perfectly valid file that is simply slow to query, and nothing but inspection distinguishes it from a good one.