Why architecture matters here
Spark SQL performance is often optimizer performance. A query that runs 10x faster after an upgrade usually got new rules or better stats. A query that regressed usually lost a physical strategy or a stat.
The architecture matters because tuning happens at multiple layers: schema + stats (feed the cost model), hints (nudge the physical planner), AQE (adapt at runtime), and extensions (custom rules for specialized cases).
With the pipeline in mind, you can read explain output and understand why the plan looks as it does — the first step to improving it.
The architecture: every piece explained
The top strip is the semantic path. SQL / DataFrame lands as a logical plan tree. Analyzer resolves column references and types against the catalog. Optimizer applies rule-based rewrites — predicate pushdown, constant folding, projection pruning. Cost model uses statistics to choose join strategies and reorder joins.
The middle row is the execution path. Physical planner picks concrete operators (broadcast vs shuffle hash join, sort merge). Code gen (Tungsten) generates JVM bytecode for whole stages so tight loops run without virtual dispatch. AQE adapts at runtime — recognizes skew, coalesces small partitions, switches join strategies based on observed sizes. Executor runs the plan.
The lower rows are practice. Extension points allow custom rules and strategies for domain-specific operators. Observability centers on the explain output and query plan UI. Ops handles statistics collection, hints, and plan review for regressions.
End-to-end flow
End-to-end: a join query lands. Analyzer resolves tables + columns. Optimizer pushes filters below the join, prunes projections, reorders joins using stats. Cost model picks broadcast join for a 4 GB dimension. Physical planner generates a SortMergeJoin for the fact-fact join. Tungsten generates bytecode for the tight filter loop. Execution begins; AQE observes skew on the fact table; splits skewed partition. Coalesces small post-shuffle partitions. Total runtime 12 seconds vs 40 without AQE and codegen. Explain output shows all rules applied.