Why it matters

Understanding planning helps diagnose slow queries. A query that would be fast if not for a bad join order is a planner issue, not a data issue.

Advertisement

The architecture

Parse and analyze work the same as any SQL engine. Table and column references are resolved against the Catalog cache.

Single-node planning generates a tree of relational operators: scans, filters, projections, joins, aggregates. The cost-based optimizer picks join orders based on statistics.

Distributed planning takes the single-node plan and inserts exchange operators. Each contiguous block of operators without an exchange becomes a fragment.

Impala planning phasesParse+analyzeAST → resolvedSingle-node planlogical → physicalDistributed planadd exchange opsFragments distributed to Impalads holding relevant data blocks
Four-phase Impala planning.
Advertisement

How it works end to end

Fragments are scheduled onto Impalads. Scan fragments run on Impalads colocated with the target HDFS blocks. Shuffle fragments run on any Impalad. Broadcast fragments run wherever the broadcast side lives.

Stats matter enormously. Without accurate stats, the CBO guesses join orders wrong, producing plans that shuffle huge tables when a broadcast would have worked. COMPUTE STATS keeps stats fresh.

Runtime filters (bloom filters computed at runtime from join keys) push down join predicates into scans, reducing data volume dramatically.