Core concept

External shuffle service for stability. AQE dynamically adjusts.

Advertisement

How it works

Too many shuffle partitions = overhead. Too few = spill.

Advertisement

Trade-offs + gotchas

Every non-trivial job.