Why it matters
Wrong partition count is the top cause of Spark performance surprises. Too few partitions and you leave cores idle. Too many and task overhead dominates. Finding the right count is workload-specific.
The architecture
Stage construction: the DAG scheduler identifies which transformations require a shuffle. Wide dependencies (each parent partition contributes to multiple child partitions) form stage boundaries. Narrow dependencies (one-to-one or one-to-few) chain within a stage.
Stage types: ShuffleMapStage produces shuffle output for downstream stages. ResultStage produces the final action result.
How it works end to end
Task = 1 partition × 1 stage. If stage has 200 partitions, it runs 200 tasks. If cluster has 100 cores, up to 100 run in parallel; the rest wait.
Partition count is set by the source (spark.sql.files.maxPartitionBytes for files), by shuffle output (spark.sql.shuffle.partitions), or explicitly via repartition/coalesce.
AQE dynamically coalesces small partitions after shuffle. This handles the common problem of shuffle producing many tiny partitions.