Why it matters
Every Spark performance discussion eventually comes back to shuffle. Reducing shuffle bytes, avoiding skew during shuffle, and choosing between shuffle join and broadcast join together are 80 percent of Spark tuning.
The architecture
The shuffle write phase: each map task partitions its output by hash-modulo of the shuffle key. Partitioned output is written to disk in a compact binary format (SortShuffleWriter or BypassMergeSortShuffleWriter depending on partition count).
The shuffle read phase: each reduce task starts by fetching its partition from every mapper's output. Fetching happens in parallel; data is buffered in memory (spilling to disk on overflow) before being processed.
How it works end to end
External shuffle service runs on each worker independent of executors. Executors can die without losing shuffle files, since the service continues serving them. Always enable this in production.
Push-based shuffle (Magnet) is a newer optimization where mappers proactively push data to reducer-side merger services, reducing many-to-many fetch overhead.
Skewed keys concentrate shuffle load on one reducer. AQE detects skew and splits the skewed partitions. Salting the join key is a manual alternative.