Why architecture matters here
The economics of a warehouse query are lopsided. Reading and decompressing bytes off HDFS or object storage, transferring them across the network, and deserializing them into row objects consumes the overwhelming majority of wall-clock time; the actual comparison amount > 1000 is nanoseconds. This means the winning move is never to make the comparison faster — it is to never read the bytes the comparison would reject. Predicate pushdown is the mechanism that converts a logical filter into physical avoidance of I/O, and it is why two syntactically identical queries against the same data can differ by two orders of magnitude in cost depending on table layout and statistics.
Pushdown operates at three distinct granularities, and understanding them as a hierarchy is the key insight. The coarsest is partition pruning: if the table is partitioned by dt and the predicate constrains dt, the planner eliminates whole directories before any file is opened — the cheapest possible skip because it never touches storage. The middle granularity is columnar block skipping: within a surviving ORC or Parquet file, each stripe or row group carries min/max statistics per column, so a reader can skip a 64MB stripe whose amount range is [0, 50] without decompressing it. The finest is value-level skipping via dictionaries and bloom filters, which can reject an entire block for an equality predicate on a high-cardinality column.
These granularities compound multiplicatively. Partition pruning might cut the data to 1/365th; stripe skipping might cut the survivors to 1/10th; bloom filters might cut those to 1/5th. The result is a query that logically scans a year of data but physically reads a few hundred megabytes. Crucially, each layer depends on metadata being present and correct — partitions registered in the metastore, ORC indexes written, statistics current. When any layer's metadata is stale or missing, pushdown at that granularity silently disables and the query falls back to reading everything below it.
The second reason architecture matters is correctness under complexity. Not every predicate is safe to push. A predicate that calls a non-deterministic UDF, references a column produced by a computation, or straddles a nullability boundary in a way the reader handles differently than the SQL engine must stay in the engine, evaluated on fully materialized rows. Hive's compiler therefore splits a compound predicate into a pushable part and a residual part, pushes the former, and keeps the latter as a normal Filter operator. Getting that split wrong in either direction — pushing something unsafe, or failing to push something safe — is the difference between a correct fast query and either a wrong answer or a slow one.
The architecture: every piece explained
Start at the top of the diagram. The HiveQL WHERE clause is parsed into an expression tree of predicates combined with AND/OR. The parser and cost-based optimizer normalize this into a logical plan and, critically, apply predicate transitivity and constant folding — if a join equates two columns and one side has a literal filter, the CBO can synthesize the equivalent filter on the other side, creating pushdown opportunities the user never wrote. The optimizer also decides join order using the same column statistics that pushdown relies on, which is why stats quality affects both.
The predicate splitter is the heart of the safety logic. It walks the predicate tree and classifies each leaf: is it deterministic? Does it reference only base-table columns (not derived expressions)? Is the comparison one the storage reader understands (equality, range, IN, IS NULL)? Pushable leaves are collected; anything referencing a UDF, a computed column, or an unsupported operator is retained as residual. For an AND of predicates, Hive can push the pushable conjuncts and keep the rest; for an OR, it can only push if every branch is pushable, because a partially-pushed disjunction would incorrectly drop rows.
The pushdown planner then decides the target layer for each pushable predicate. Predicates on partition columns become partition pruning directives evaluated against the metastore — Hive lists only the matching partition directories, so a year-partitioned table with a single-day filter opens one directory. Predicates on data columns are compiled into a SearchArgument (SARG): a serialized, format-neutral representation of the filter (AND/OR/NOT of leaf comparisons) that is handed to the ORC or Parquet reader via the input format. The SARG is the contract between Hive and the storage layer.
Inside the reader, the SARG drives stripe and row-group skipping. ORC stores a file footer and per-stripe indexes with min/max (and optionally bloom filters) for each column; the reader evaluates the SARG against those statistics and skips any stripe whose statistics prove no row can match. Within a surviving stripe, row-group-level indexes (every 10,000 rows) allow finer skipping. Parquet does the analogous thing with row groups and column chunk statistics. Where bloom filters or dictionaries exist, equality predicates get an extra, cheaper rejection test before any decompression. Whatever the reader could not prove — the residual filter — runs as a normal operator on the rows that survive, guaranteeing the final result is exactly correct regardless of how aggressive the skipping was.
The bottom strip is the operational reality that makes all of this actually fire: column statistics must be collected and fresh, files must be written with useful sort orders and index strides, and predicates must be written in a form the reader can recognize. Every one of those is a knob a data engineer controls, and the diagram's arrows only carry data efficiently when those knobs are set correctly.
End-to-end flow
Trace SELECT customer_id, amount FROM transactions WHERE dt = '2026-07-01' AND status = 'FAILED' AND amount > 1000 against a table partitioned by dt, stored as ORC, sorted by status, with a bloom filter on status and current column statistics. The transactions table holds two years of daily partitions and eight terabytes total.
Compilation: the CBO parses the three conjuncts. The splitter classifies all three as deterministic base-column predicates, all pushable. The planner routes dt = '2026-07-01' to partition pruning and combines status = 'FAILED' and amount > 1000 into a SARG for the ORC reader. Partition pruning runs first, against the metastore: of 730 partitions, exactly one matches, so Hive lists only /warehouse/transactions/dt=2026-07-01/. The scan drops instantly from eight terabytes to roughly eleven gigabytes — the single day.
Split generation: the query planner splits that day's ORC files into input splits mapped to map tasks. Each task opens its file's footer and reads stripe statistics. Stripe skipping: because the file is sorted by status, the FAILED rows cluster into a handful of contiguous stripes; stripes whose status min/max range excludes FAILED are skipped without decompression. The bloom filter on status confirms membership at the row-group level, rejecting groups that happen to straddle the boundary. The amount min/max further skips any surviving stripe whose amounts top out below 1000.
What survives — perhaps two hundred megabytes of stripes that genuinely might contain matching rows — is decompressed and deserialized. The residual filter applies status = 'FAILED' AND amount > 1000 exactly on those materialized rows, because min/max skipping is conservative (it can only prove absence, never presence). The final rows flow to the projection, which reads only the customer_id and amount columns — columnar projection is itself a form of pushdown, avoiding I/O on the dozens of columns the query never mentions.
Now imagine one thing broken: the table's column statistics are stale because a bulk load never ran ANALYZE. Partition pruning still works (it uses the metastore's partition list, not column stats), so dt pruning survives. But the CBO, lacking accurate status and amount cardinalities, may pick a worse join order elsewhere, and — depending on ORC index presence — stripe skipping still works because ORC statistics live in the file footer, not the metastore. This distinction matters operationally: partition pruning depends on metastore partition registration, the CBO's plan quality depends on ANALYZE stats, and block skipping depends on indexes written into the file at ingest. Three separate metadata sources, three separate failure modes, one query.