Why architecture matters here
Architecture matters here because the gap between the JVM engine and the hardware's true capability is large, and it shows up exactly on the workloads that cost the most: big joins, aggregations, and scans over wide columnar tables. The JVM processes one row at a time through generated code; each row pays interpretation, bounds-checking, and virtual-dispatch overhead, and the CPU's vector units sit mostly idle. A vectorized engine flips this: it processes a batch of thousands of values with a single tight loop, feeds the SIMD units, keeps data in cache, and amortizes per-operation overhead across the whole batch. On CPU-bound analytical queries that difference is routinely 2-4x.
The second reason is garbage collection. A JVM engine under heavy shuffle and aggregation allocates enormous numbers of short-lived objects, and GC pauses inject latency spikes and steal CPU that should be doing query work. Photon manages its memory manually in off-heap buffers: no allocation churn, no stop-the-world pauses, and precise control over spilling when memory runs low. For long-running jobs the elimination of GP jitter is worth as much as the raw vectorization in predictability terms.
But the architecture also has to earn its keep on mixed queries, and this is the subtle part. Real pipelines contain unsupported functions, exotic types, and custom code. A native engine that could only run pure-native queries would rarely fire. Photon's value depends entirely on partial coverage being safe and cheap: it must execute the native fraction, hand the rest to the JVM, and keep the row-to-columnar transitions from eating the speedup. Understanding where those transitions live is what separates a query that gets 3x from one that Photon technically 'ran' but barely accelerated.
There is also a compatibility discipline that the architecture must uphold, and it constrains the design more than raw speed does. Photon has to produce the same results Spark's JVM engine would across every supported type, null-handling rule, and overflow behavior, because the same query may run partly native and partly on the JVM within one plan and the two halves must agree. That is why Photon does not simply reimplement a fast subset with its own convenient semantics; it mirrors Spark's semantics exactly and falls back rather than risk divergence. The payoff for users is that enabling the engine is safe — correct queries stay correct — and the cost is that native coverage grows conservatively, one carefully-validated operator and type at a time. Knowing this explains why coverage on your workload is empirical rather than guaranteed: it depends on which operators and types your queries actually use.
The architecture: every piece explained
Top row: planning and dispatch. Spark's Catalyst optimizer produces the physical plan exactly as it always has — Photon does not replace the optimizer. A Photon planner pass then walks that physical tree and tags each operator as native-capable or not, based on the operator type, the expressions inside it, and the data types involved. Supported subtrees become native operators (C++ vectorized kernels for scans, filters, projections, hash aggregates, hash joins, sorts); everything else stays as JVM operators. The plan is thus a hybrid tree, and the split is decided per query, per operator.
Middle row: the data plane. Photon does not move rows; it moves columnar batches — vectors of values in an Arrow-like layout, typically a few thousand rows wide, held in contiguous off-heap buffers. SIMD expression kernels evaluate predicates and projections over a whole batch at once: a filter computes a selection vector, a projection applies an arithmetic kernel across the column, and branches are minimized so the CPU's pipeline and vector units stay busy. Adaptive execution still runs on top: Photon cooperates with AQE so shuffle partition counts, skew handling, and join-strategy switches are decided at runtime from real statistics, and Photon executes the resulting stages natively.
Bottom row: the bridges and memory. Off-heap memory is managed by Photon itself in manual buffers — no JVM heap, no GC — with an explicit budget and a spill path to disk when a hash table or sort exceeds it. Transition operators are the seams of the hybrid plan: where a native subtree feeds a JVM operator (or vice versa), a transition converts columnar batches to rows or rows to columnar. Each transition costs CPU and memory, so the planner tries to place them at natural stage boundaries and to keep native subtrees large so transitions are few.
The ops strip reflects what actually determines your speedup. Photon coverage percentage is the fraction of the query's work that ran natively. Transition overhead is what the row-columnar conversions cost. Off-heap pressure and spill rate tell you whether Photon's memory budget is adequate or whether it is falling back to disk. These are the numbers to read when a query 'used Photon' but did not get faster.
One more architectural subtlety deserves attention: the columnar batch width is a real tuning surface. Batches that are too narrow spread the fixed per-batch overhead across too few rows and leave the SIMD units underfed; batches that are too wide blow past the CPU cache and start paying memory-bandwidth stalls instead of running cache-resident. Photon picks a batch size in the low thousands of rows to sit in that sweet spot, but the effective width also depends on row size, so very wide schemas with many large columns behave differently from narrow numeric tables. This is invisible in the query text and visible only in profiling, which is why the operational habit of reading the plan and the runtime metrics — rather than trusting that native equals fast — matters as much for Photon as it does for any vectorized engine.
End-to-end flow
Trace a query: read a large fact table from Parquet, filter by date, join to a dimension table, aggregate a sum grouped by region, and finally apply a Python UDF to format the output. Catalyst compiles the physical plan; the Photon planner tags the Parquet scan, the filter, the hash join, and the hash aggregate as native, and tags the Python UDF as JVM-only because Photon does not execute arbitrary Python.
Execution begins in native code. The vectorized Parquet reader loads column batches directly into off-heap buffers — no row materialization. The filter kernel evaluates the date predicate across each batch and produces a selection vector, skipping non-matching rows without branching per row. The surviving batches flow into the native hash join: the dimension table is built into an off-heap hash table, and probe batches are matched with SIMD-friendly lookups. All of this stays columnar, cache-resident, and GC-free.
The hash aggregate accumulates the grouped sums natively, maintaining its aggregation hash table off-heap. If the number of groups grows large enough to exceed the memory budget, Photon spills partitions to disk and merges them — the same logical operation as the JVM engine but with manual buffer control and no allocation storms. AQE, watching the real shuffle statistics, may coalesce the post-shuffle partitions or switch a skewed join to a broadcast; Photon executes whatever plan AQE lands on.
Now the boundary. The aggregate's native columnar output must feed the Python UDF, which runs on the JVM path. A transition operator converts the columnar batches to rows and hands them across; the UDF executes, and its results flow on to the output. Because this transition sits at the very end of the plan on a small, already-aggregated result, its cost is trivial — the expensive scan-filter-join-aggregate spine ran entirely native. Had the UDF sat in the middle, forcing row-columnar transitions around the big join, the picture would be very different, and the coverage and transition-overhead metrics would show it. This is why the same feature can 3x one query and barely move another: placement of the unsupported operator, not its mere presence, decides the outcome.
It helps to picture the vectorized inner loop to see where the speed comes from. In the JVM engine, the generated code for the date filter loops over rows, and for each row loads a value, tests the predicate, branches, and conditionally appends to the output — a branch the CPU cannot predict well when the data is mixed, and a per-row overhead that dwarfs the actual comparison. Photon's kernel instead loads a contiguous run of date values into a vector register, compares the whole vector against the bound in a single SIMD instruction, and writes a packed selection mask, with no per-row branch at all. Multiply that pattern across the filter, the join's hash probe, and the aggregate's accumulation, and the compounded saving is the 2-4x that shows up on CPU-bound analytical queries — provided, as this trace shows, the transitions to the JVM path stay off that hot spine.