Why it matters

ESS is prerequisite for dynamic allocation. Understanding shapes efficient Spark.

Advertisement

The architecture

Auxiliary process on each worker.

Holds shuffle files.

Executors write to ESS; readers pull from ESS.

External shuffle serviceExecutor writesshuffle filesESS on workerholds filesReader executorpulls from ESSEnables executor removal without shuffle recompute
ESS role.
Advertisement

How it works end to end

Deploy: on YARN as NodeManager aux service; on K8s as separate pods.

Enables: dynamic allocation of executors.

Disk: shuffle files stored on worker disk.

Alternatives: Push-Based Shuffle (Magnet); Celeborn.