Why architecture matters here
The reason codegen matters is the cost model of interpretation at scale. Consider evaluating the predicate WHERE l_quantity > 30 AND l_shipdate < '1998-09-01' over a billion-row table. An interpreted engine represents that expression as a tree of objects; evaluating it per row means walking the tree, dispatching a virtual call at each node to the right operator, checking null flags, and reading the operand types from metadata — dozens of instructions and several unpredictable branches per row, most of them re-deciding facts that are constant for the whole query. Multiply by a billion rows and the interpretation overhead, not the actual comparison, dominates the CPU time.
Codegen attacks this by exploiting what is known at query-compile time but not at engine-build time. The column types are fixed once the query is planned; the constants 30 and the date are literals; the tuple layout is known. A query-specific compiled function can therefore hardcode the types (no runtime type switch), inline the constants (the comparison becomes a direct machine instruction against an immediate), fold away null checks for columns declared NOT NULL, and eliminate the virtual dispatch entirely by inlining the operator bodies. What was a tree walk with indirect calls becomes a tight, branch-lean loop the CPU can pipeline and vectorize.
Two properties make this especially valuable for an MPP analytical engine. First, the work is loop-dominated: the same inner function runs over enormous row counts, so removing even a few instructions per row compounds massively — this is the opposite of a transactional workload where each query touches few rows and compilation would never pay back. Second, the engine is written in C++, so the runtime functions codegen wants to call and inline can be cross-compiled to the same LLVM IR the generated code lives in, letting the JIT inline across the boundary between hand-written and generated code. Without that, generated code could only make opaque calls into the runtime and would lose most of the benefit. The combination — known-at-plan specialization plus cross-module inlining — is what turns codegen from a micro-optimization into a first-order performance feature.
The architecture: every piece explained
The foundation is cross-compilation. Impala's performance-critical C++ runtime functions — expression evaluators, the tuple/slot accessors, hash functions, the aggregation update routines — are compiled twice: once by the normal C++ compiler into the binary, and once by Clang into LLVM bitcode that ships inside the impalad. That bitcode is the raw material codegen links against, so when the engine generates a specialized function it can pull in the actual body of a runtime routine and let LLVM inline and optimize it together with the generated logic, rather than emitting a call across an opaque boundary.
At query start, the codegen module builds LLVM IR for the hot parts of each plan fragment. It does not compile the whole engine — only the operators where per-row work dominates: scan slot materialization, WHERE and HAVING expression evaluation, the build and probe of hash joins, aggregation, and sorting comparators. For each, it constructs a function tailored to the query: the column types are baked in, literal constants are emitted as immediates via constant propagation, null-handling is specialized to the actual nullability, and calls to runtime helpers are wired to the cross-compiled bitcode so they can be inlined.
The assembled IR is handed to the LLVM JIT, which runs an optimization pipeline — inlining, constant folding, dead-code elimination, instruction combining — and emits native machine code for the host CPU. The result is a set of specialized native functions that replace the generic interpreted evaluators. Because emitting them is expensive, Impala addresses the latency two ways: it keeps codegen optional per query (skipping it when the planner estimates the query is too small to benefit), and it compiles asynchronously — execution begins on the interpreted path immediately, and the runtime swaps in the native functions via function pointers once compilation completes, so compilation latency overlaps with real work rather than stalling the query.
Around this sit the operational surfaces. The decision to codegen is influenced by the query's estimated cardinality and can be forced or disabled with the DISABLE_CODEGEN and DISABLE_CODEGEN_ROWS_THRESHOLD query options. The query profile reports codegen time separately (the 'CodeGen' phase, with compile time per fragment), which is the primary diagnostic for deciding whether codegen is helping or hurting a given workload. And because the JIT-compiled code targets the specific host CPU, it can exploit available instruction-set extensions.
It is worth being precise about what gets compiled, because a common misconception is that codegen compiles 'the query.' It does not compile the SQL, the planner, or the control logic that orchestrates operators; those remain interpreted C++ and are a negligible fraction of the cost on a large scan. Codegen targets specifically the per-tuple inner functions — the code that runs once per row — because that is where interpretation overhead is multiplied by the row count. A hash join, for example, gets a generated function to compute the join key and hash from a probe row and another to compare against build-side keys, both specialized to the exact key columns; the surrounding logic that manages spill partitions and iterates batches stays interpreted. This selectivity is what keeps compilation cost bounded — the engine generates a handful of small, hot functions per fragment rather than a whole program — and it is why the payoff scales with rows processed but the compilation cost scales with query complexity, making the ratio between them the thing that decides whether codegen wins.
End-to-end flow
Trace a large aggregation query: SELECT l_returnflag, SUM(l_quantity) FROM lineitem WHERE l_shipdate < '1998-09-01' GROUP BY l_returnflag over a billion-row table. The planner produces fragments — scan, filter, partial-aggregate, and a final aggregate — and estimates the scan will process enough rows that codegen will pay off, so codegen is enabled for this query.
As each executor fragment starts, its codegen module builds IR. For the scan's filter it generates a function specialized to the l_shipdate < date predicate: the date literal is inlined as an immediate, l_shipdate's type and slot offset are hardcoded, and the comparison compiles to a couple of native instructions with the runtime's date-decoding helper inlined from bitcode. For the aggregation it generates a specialized update function that knows the grouping key is a single small column and the aggregate is a SUM over a specific numeric type, eliminating the generic type-dispatch the interpreted aggregator would perform per row.
Compilation of these functions takes, say, 60ms per fragment. Rather than block, the fragment begins executing on the interpreted evaluators immediately, streaming the first row batches through the generic path. When the LLVM JIT finishes emitting native code, the runtime atomically swaps the function pointers the inner loop calls, so from that point on every batch runs through the specialized native filter and aggregator. The 60ms of compile time overlapped with the first fraction of a second of scanning, and the remaining hundreds of millions of rows run several times faster than they would have interpreted. The net query time drops well below the interpreted baseline despite paying for compilation.
Now the counterexample the architecture must handle. A dashboard fires SELECT * FROM small_dim WHERE id = 42, a point lookup returning a handful of rows. Codegen here would spend tens of milliseconds compiling functions that then run over almost no rows — pure overhead. Because the planner's row estimate is below the codegen threshold, Impala skips codegen entirely and runs interpreted, finishing in single-digit milliseconds. The same mechanism, driven by the cost estimate, both accelerates the billion-row aggregation and avoids sabotaging the tiny lookup — and if the estimate is wrong, an operator can read the CodeGen phase in the profile and override the behavior with query options for that workload.
The asynchronous, function-pointer-swap design deserves a closer look because it is what makes the whole trade-off practical. If compilation were synchronous, every codegen'd query would stall at startup for the full compile time before processing a single row, and the latency would be visible and painful. By running the interpreted path immediately and swapping to native code the instant it is ready, Impala hides the compile latency behind useful work: the rows scanned during those first tens of milliseconds are processed by the generic evaluators, and everything after the swap runs native. For a query that processes billions of rows over several seconds, the interpreted prefix is a rounding error and the swap is effectively free; for a query that finishes in less time than compilation takes, the swap may never even happen before the query completes, which is precisely the case the row-threshold heuristic tries to avoid by not compiling at all. The swap is atomic at the function-pointer level, so there is no torn state — a batch is processed entirely by one implementation or the other, never half by each — and correctness is identical between the two paths by construction.