Why architecture matters here
Memory is the scarcest and most contended resource in a Spark executor, and the framework's whole performance model is built around not running out of it while not wasting it. The reason a unified design beats fixed pools is that real workloads are unbalanced in unpredictable ways: an ETL job shuffles heavily and caches nothing; an iterative ML job caches a training set and reuses it across dozens of passes; an interactive query caches some tables and shuffles others. Fixed pools force you to guess the ratio in advance and pay for guessing wrong — idle storage memory while execution spills, or idle execution memory while cache evicts. The unified region lets the actual workload determine the split moment to moment, which is strictly better whenever the workload isn't known in advance, which is almost always.
The eviction asymmetry is the load-bearing design decision and it follows from a correctness argument, not a performance one. Cached data is, by definition, derived — it can be recomputed from its lineage if it's gone. Execution data (the in-progress hash table of a join, the sort buffer of an aggregation) is not derived; it is the computation itself, and if it can't get memory the task fails. So the rule writes itself: when the two compete, execution wins and storage yields, because yielding storage costs a recomputation while yielding execution costs a crash. This is why aggressively caching a huge dataset doesn't protect it — a later shuffle-heavy stage will evict it right out from under you, and the query that was 'cached' silently recomputes.
Spilling is the pressure valve that makes the whole thing survivable rather than brittle. Without spill, any operation whose working set exceeded execution memory would simply fail; with spill, a sort or hash aggregation that outgrows memory serializes sorted runs or hash partitions to local disk and merges them back, completing correctly but slower. This converts the failure mode from 'crash' to 'degrade,' which is enormously more operable — but it also means that heavy spilling is the signature of a job running near its memory limit, and the metric to watch. A job that spills tens of gigabytes isn't broken, but it is telling you that its partitions are too big, its memory share too small, or its data too skewed. The practical corollary is that spill bytes are a tuning dial, not a failure alarm: a little spill is the price of running near capacity, while sustained multi-gigabyte spill on every stage is the system asking you to repartition or add memory before the next, larger input tips it from degrade into crash.
Understanding the layout also explains the OOMs that spilling doesn't prevent, and those are the ones that actually kill jobs. Spill protects the managed execution region; it does nothing for memory Spark doesn't manage — the objects your UDF allocates, an oversized broadcast, or a giant result collected to the driver. And a single partition so large that even its spilled runs can't be processed will OOM despite spilling being enabled. So the diagnostic discipline is to ask which region is under pressure: managed execution (tune partitions and memory fraction), user memory (fix the UDF or reduce object churn), or the driver (stop collecting). The layout is the map for that diagnosis.
The architecture: every piece explained
Top row: how the heap is carved. An executor is a JVM process with a heap sized by spark.executor.memory. Spark first sets aside a small reserved region (about 300MB) as a safety margin so the internals always have room. What remains is split by spark.memory.fraction (default around 0.6): the fraction inside becomes the unified region that Spark manages for execution and storage, and everything outside is user memory — the space for the objects your code creates, the data structures inside your UDFs and closures, and anything Spark doesn't track. User memory is deliberately unmanaged: Spark won't spill or evict it, which is exactly why a UDF that builds a huge in-memory map is a classic OOM even on an executor with plenty of 'Spark' memory free.
Middle row: the unified region at work. Inside it, execution memory serves the machinery of query processing — shuffle buffers, the hash tables of hash joins and hash aggregations, and sort buffers — while storage memory holds cached/persisted blocks and broadcast variables. The borrow-and-evict boundary between them is dynamic: if execution is idle, storage can expand to use the whole region; when execution needs memory, it reclaims space, evicting cached blocks if necessary down to a small protected storage floor. When execution needs more than eviction can free, it spills to disk: the sort or hash operator writes its current contents out as serialized runs and continues, merging spilled data back at the end. Storage never spills execution — that is the asymmetry made concrete.
Bottom rows: the format and the levers. Tungsten is why Spark's memory goes further than naive JVM objects would: it stores rows in a packed, off-heap-capable binary layout that avoids per-object overhead and GC pressure, so a shuffle or aggregation manipulates compact byte regions rather than millions of boxed Java objects. Off-heap execution memory (spark.memory.offHeap) moves this out of the JVM heap entirely to cut GC. The storage level chosen when you persist — MEMORY_ONLY, MEMORY_AND_DISK, the _SER serialized variants — decides whether evicted blocks are dropped (and recomputed) or written to disk, and whether they're stored as objects or compact serialized bytes. The ops strip is the diagnostic surface: spill bytes and GC time in the stage metrics, cache size and eviction in the storage tab, partition sizes, and the OOM signatures that tell you which region failed.
End-to-end flow
Trace a job that joins a large fact table to a medium dimension and caches the result for repeated queries. The executor has 8GB: ~300MB reserved, ~4.6GB unified region (0.6 of the rest), the remainder user memory. The stage begins with plenty of free unified memory.
First the job caches the dimension table with persist(MEMORY_AND_DISK). Storage memory fills with those blocks; because execution is idle at this point, storage is allowed to borrow well into the region. Queries against the cached dimension are fast — reads come straight from storage memory, no recomputation. So far storage owns most of the unified region and nothing is under pressure.
Now a heavy shuffle-join stage starts. Its hash-join build side and shuffle buffers demand execution memory, and the unified region is mostly occupied by cached blocks. Execution exercises its privilege: it reclaims space from storage, evicting cached dimension blocks down toward the protected storage floor. The blocks were persisted MEMORY_AND_DISK, so evicted ones spill to local disk rather than vanishing — later reads pay a disk hit instead of a full recompute. The join's hash table grows; when it exceeds the execution memory it can hold even after eviction, the operator spills sorted/partitioned runs to disk and merges them back. The stage completes correctly, the Spark UI shows tens of gigabytes of spill and elevated GC time, and the job is slower than it would be with more memory or smaller partitions — but it does not crash. This is the system working as designed: degrade, don't die.
The stress case is skew plus unmanaged memory, the two things spill can't save. One join key is enormously hot, so a single partition is ten times the size of the others; its hash table is too large to spill effectively (the spilled runs for that one key still overwhelm the operator), and that task OOMs while every other task finishes fine — the classic 'one straggler killed the stage' signature that points at skew, not total memory. Separately, a UDF in the pipeline builds a big lookup map per partition in user memory; Spark neither tracks nor spills it, so on a partition with many distinct keys the executor OOMs with the unified region half empty. Both crashes look like 'not enough memory,' but the fix for the first is salting or AQE skew handling and smaller partitions, and the fix for the second is rewriting the UDF or enlarging user memory — and only the region-by-region map tells you which. Bumping spark.executor.memory blindly might paper over one and do nothing for the other.