Why architecture matters here
The architecture matters because the alternative extremes both fail on a shared cluster. Pure first-come-first-served lets a single large job that arrives first monopolize the cluster for hours while everyone else waits — technically simple, operationally intolerable. Static partitioning gives each team a fixed slice, which prevents monopolization but wastes enormous capacity whenever a team's slice sits idle while another team is starved next to it. The Fair Scheduler threads between them: it prevents any one job from monopolizing by steering toward fairness, but it keeps the cluster fully utilized by lending idle capacity. That combination — no starvation and no waste — is the whole reason to run it, and it is genuinely hard to get any other way.
It matters because fairness has to be dynamic to be useful. A fair share computed against the configured queues would leave capacity idle whenever some queues have no work; the Fair Scheduler instead computes fair share against active demand, so the instantaneous fair slice of each busy queue grows as others go idle and shrinks as they return. This is what lets a single overnight batch job use the entire cluster when nothing else is running, then gracefully cede half of it the moment the morning's interactive queries arrive. The dynamic recomputation is not a detail; it is the mechanism that reconciles high utilization with fairness.
The architecture also matters because lending is only safe if reclamation is reliable, and reclamation means preemption. If idle capacity is lent to a greedy queue and there is no way to take it back, a returning queue can be starved for as long as the borrower holds its containers — which for long-running tasks can be a very long time. Preemption is the enforcement arm of fairness: when a queue sits below its fair (or minimum) share for longer than a timeout, the scheduler kills over-share containers to make room. This is powerful and dangerous in equal measure — reclaimed work is lost and must rerun — so the architecture has to bound preemption carefully, which is why timeouts and thresholds are first-class configuration rather than hidden constants.
Finally, it matters because a cluster is not one resource but two — memory and CPU — and fairness measured wrongly across them produces absurd allocations. If you balanced only on memory, a CPU-bound queue could grab all the cores while looking fairly treated on the memory axis, and vice versa. Dominant Resource Fairness fixes this by identifying each user's dominant resource (the one they consume the largest fraction of) and equalizing dominant shares, so a memory-heavy and a CPU-heavy job are balanced by the resource each actually cares about. Without DRF, multi-resource fairness degenerates; with it, the intuitive notion of 'a fair slice' survives the fact that jobs are shaped differently.
The architecture: every piece explained
Top row: from submission to assignment. Applications submit into leaf queues of a hierarchy — a tree of queues, each with a weight that sets its relative entitlement and a scheduling policy (fair, FIFO, or DRF) governing how it divides its share among its own apps. The scheduler continuously runs a fair share calculation: given the current set of active queues and apps and their weights, what fraction of the cluster does each one deserve right now? When a node reports a freed container, the scheduler assigns it to whichever queue and app is furthest below its fair share — the simple greedy rule that, applied repeatedly, drives the whole cluster toward the fair allocation.
Middle row: lending and reclamation, the dynamic heart. When some queues are idle, their unused capacity is lent to busy queues so the cluster stays full rather than holding capacity in reserve. But lending creates a debt: when an idle queue's demand returns and it finds itself below its fair share, the scheduler must reclaim capacity. If waiting for natural container turnover is too slow, preemption kicks in — the scheduler selects and kills containers from queues that are over their fair share to free room for the starved queue. The result is a rebalanced cluster that converges back toward fairness. The whole cycle — lend when idle, reclaim when demand returns — is what keeps utilization high without letting borrowers permanently starve owners.
Bottom-left: multi-resource fairness. Because a container bundles memory and CPU, and jobs weight those differently, the scheduler uses Dominant Resource Fairness: for each user it computes the fraction of each resource they hold, takes the maximum (their dominant share), and equalizes dominant shares across users. A memory-bound job is balanced on memory, a CPU-bound job on CPU, and the two coexist fairly instead of one silently dominating the axis the other ignores.
Bottom-right and ops: bounds and levers. Fair share is the default target, but two bounds override it: minShare guarantees a queue a floor of resources it can always reclaim (so a critical production queue never starves), and maxShare caps a queue's ceiling (so a greedy queue cannot swallow the cluster even when others are idle). The ops strip collects the configuration that shapes behavior: queue weights for relative entitlement, minShare guarantees for critical queues, preemption timeouts that control how patiently the scheduler waits before reclaiming, the DRF policy for multi-resource fairness, and placement rules that decide which queue an incoming app lands in based on user, group, or specified queue.
End-to-end flow
Trace a shared cluster through a full day — an overnight batch job, the morning interactive rush, a starved production queue, and the preemption that rescues it.
Night — one queue, whole cluster: at 2am only the etl queue has work: a large Spark pipeline. With no other active queues, the Fair Scheduler computes the pipeline's fair share as essentially the entire cluster and lends it all the idle capacity. The job runs at full width and finishes far faster than any static slice would have allowed. This is dynamic fairness paying off on the upside: idle capacity is never wasted just because it is nominally allocated elsewhere.
Morning — demand arrives, shares shrink: at 9am analysts start submitting Hive and Spark queries into the adhoc queue, and the production reporting queue kicks off its scheduled jobs. Now three queues are active. The scheduler recomputes fair shares against live demand: each queue's fair slice drops toward its weighted portion of the cluster, and as etl containers finish, the freed ones are assigned to adhoc and reporting because they are furthest below their fair shares. Within a few scheduling cycles the cluster drifts from all-etl toward the three-way weighted split, without anyone intervening.
Starvation risk — long tasks hold the line: the trouble is that etl's remaining containers are long-running, so natural turnover is slow. The reporting queue, which has a minShare guarantee because it feeds business dashboards, is sitting below both its fair share and its minShare, waiting for containers that etl is not releasing fast enough. Fairness on paper is not fairness in practice while the over-share queue clings to its containers.
Preemption — reclamation with a bounded blast: once reporting has been below its minShare for longer than the configured preemption timeout, the scheduler acts. It identifies containers in etl, which is over its fair share, and preempts the minimum necessary to bring reporting up to its guarantee — signaling the ApplicationMaster first where possible so the killed tasks can be rescheduled cleanly. The reclaimed capacity flows to reporting, its dashboards refresh on time, and the cluster settles at the intended weighted split. The preempted etl tasks rerun later; the cost of that rerun is the price of the guarantee, which is why the timeout is tuned to preempt only when starvation is real, not on every transient imbalance.
Multi-resource balance underneath: throughout, DRF is quietly keeping the split sensible. The etl Spark job is memory-heavy while some adhoc queries are CPU-heavy, so the scheduler equalizes each queue's dominant resource rather than any single axis — etl is held to its memory-dominant share and the CPU-bound work to its CPU-dominant share, so neither starves the other on the resource it actually needs. The day's allocation stays fair not just in aggregate but across both resources the containers carry.