Why architecture matters here
The architecture matters because the alternative to spilling is query failure, and query failure is the worst possible outcome for a user who just wants an answer. An in-memory engine with no spill support treats memory as a hard wall: a query that needs one byte more than its limit is killed, wasting all the work it had done and forcing the user to either shrink the query or beg for more memory. Spilling converts that hard wall into a soft slope — the query slows as it crosses the memory line but keeps making progress — which is the difference between an engine that only works for queries you have pre-sized and one that works for the messy, unpredictable queries real users actually run. Graceful degradation is not a luxury; it is what makes an in-memory engine usable on unbounded data.
It matters because the partitioned operator design is what makes spilling tractable rather than catastrophic. Spilling only helps if you can evict a self-contained piece of the working set and process it later in isolation. Impala's hash join and aggregation partition their input by hash precisely so that each partition is independent: all rows that could join or aggregate together land in the same partition, so a partition can be spilled and later reread and finished without needing anything from the partitions still in memory. This is why the operators are built around hash partitioning from the start — not as an optimization, but as the structural prerequisite that lets memory pressure be resolved one partition at a time instead of all-or-nothing.
The architecture also matters because recursion is what handles the case spilling could otherwise still fail on: a single partition too big to fit even after eviction. If the build side is badly skewed and one hash partition alone exceeds memory, spilling it and rereading it does not help — it still will not fit. The answer is recursive repartitioning: when a spilled partition read back is too large, it is hash-partitioned again into smaller sub-partitions using a different hash, and the process repeats until the pieces fit. This is what makes spilling robust to data that is much larger than memory, and understanding it explains both why Impala can complete very large joins and why pathological skew can still make them slow.
Finally, it matters because spilling only works if the memory accounting beneath it is honest, which is the job of the buffer-pool reservation system. Every spilling operator reserves a minimum amount of memory it is guaranteed to keep — enough to hold at least one partition's worth of buffers so it can always make forward progress — and additional memory is used opportunistically and released under pressure. Without guaranteed reservations, an operator could spill down to nothing and deadlock, unable to hold even the single buffer it needs to process a spilled partition. The reservation system is the contract that makes 'spill until it fits' terminate successfully instead of grinding to a halt, and it is why memory limits and reservations, not just total RAM, govern whether a query can run.
The architecture: every piece explained
Top row: the spill trigger and continue. A memory-heavy operator builds its state — a hash join filling its build-side hash table, an aggregation accumulating groups, a sort buffering rows. As it grows it eventually hits its memory limit: the buffer-pool reservation is exhausted and no more memory can be acquired. Rather than fail, the operator selects a victim and spills a partition to disk — writing that partition's buffers to a scratch file and freeing the memory. It then continues in memory with the remaining partitions, repeating the spill of another victim whenever pressure returns. The build phase thus completes with some partitions in memory and others parked on disk, having stayed within its memory limit throughout.
Middle row: reading back and recursion. In the read phase — the probe side of a hash join, or the merge of a sort — the operator brings spilled partitions back to process them. For a join, it probes each spilled build partition against the matching spilled probe partition; for a sort, it merges the sorted runs. If a spilled partition, read back, is still too big to fit in memory (severe skew), the operator recursively repartitions it with a new hash into smaller pieces and processes those, repeating until each fits. The query runs slower but completes — the added disk I/O of writing and rereading spilled data is the cost — which is the whole point: it is graceful degradation versus an out-of-memory failure that would have wasted all the work and returned nothing.
Bottom-left: the memory contract. Spilling rests on buffer-pool reservations: each spilling operator reserves a guaranteed minimum of memory — enough buffers to always process at least one partition — so it can make forward progress even under maximum pressure and never spills itself into deadlock. Memory above the reservation is used opportunistically and given back when the system needs it. This guaranteed floor is what ensures the spill-until-it-fits loop terminates rather than starving.
Bottom-right and ops: where the spilled data goes and how to govern it. Spilled partitions are written to configured scratch directories on local disk, which must have enough space to hold the spilled working set and enough I/O to write and reread it without becoming the bottleneck — if scratch fills or its disks saturate, spilling itself fails or crawls. The ops strip names the levers: per-query mem_limit to bound how much memory each query may use before spilling, scratch directory capacity and I/O provisioned for the spill volume, admission control to cap concurrency so many spilling queries do not collectively exhaust memory and scratch, spill monitoring to see which queries spill and how much, and attention to partition skew, which is what turns a manageable spill into a recursive, slow one.
End-to-end flow
Trace a large hash-join query — joining a huge fact table against a build side too big for memory — from build through spill, probe, and a skew-driven recursion, then step back to the cluster view.
Build fills and hits the limit: the join starts building its hash table from the build-side input, partitioned by the join key hash into, say, sixteen partitions. As rows stream in, the partitions grow, and partway through the operator's buffer-pool reservation is exhausted — it has reached its mem_limit share. Instead of erroring, it picks the largest in-memory partition as the victim, writes its buffers to a scratch file on local disk, and frees that memory. Building continues into the remaining in-memory partitions, spilling another victim each time pressure returns, until the entire build side has been processed — some partitions resident in memory, others spilled to scratch — all within the memory limit.
Probe respects the partitioning: now the probe (fact-table) side streams in, hashed by the same join key into the same sixteen partitions. Rows whose partition's build side is still in memory are joined immediately. Rows whose partition was spilled are themselves written to a matching probe-side scratch file, because their build counterpart is on disk. After the probe stream is consumed, the operator processes the spilled partitions one at a time: it reads a spilled build partition back into memory to rebuild its hash table, then streams the matching spilled probe partition against it, emitting join results. Because the partitioning guaranteed that matching rows share a partition, each spilled partition can be finished in isolation.
Skew forces recursion: one join key is wildly popular — a null-heavy or default value — so its partition is far larger than the others and, read back, still does not fit in memory. Spilling it again would not help. Instead the operator recursively repartitions that oversized partition with a different hash function into smaller sub-partitions, spilling and processing those. If a sub-partition is still too big it recurses again. This resolves the skew at the cost of extra passes of disk I/O over the hot partition — the query completes, but that one skewed key is why it is slow, and the monitoring will show deep recursion and heavy spill on this operator.
Scratch and reservations hold it together: throughout, every spilled buffer lands in a scratch directory that was provisioned with enough space and I/O to absorb the join's spilled volume; had scratch filled, the query would have failed at the spill rather than the build, so scratch capacity is as load-bearing as memory. And the operator never spilled below its guaranteed reservation, so it always retained the buffers needed to process the next partition and never deadlocked. The query finishes correctly, slower than an all-in-memory run would have been but successfully where a non-spilling engine would have failed outright.
The cluster view — admission control: zoom out and the real risk is not one query but many. If a dozen large queries all spill at once, they collectively contend for memory and scratch I/O, and spilling amplifies the pressure rather than relieving it. Admission control is the governor: it caps how many memory-heavy queries run concurrently and queues the rest, so the cluster admits a workload it can actually complete — some in memory, some spilling — instead of over-committing into a storm of thrashing spills that makes every query slow. Spilling makes an individual query survivable; admission control makes a cluster of spilling queries survivable.