Why architecture matters here
The Capacity Scheduler matters because multi-tenant resource sharing is a political and technical problem, and the scheduler is where the organization's priorities become enforced policy. Teams don't just need compute — they need predictable compute: the production ETL must run on time regardless of what the analysts are doing; the ML team needs its GPU allocation; the ad-hoc queue shouldn't starve production. The Capacity Scheduler encodes these as queue capacities and hierarchies — the finance queue guaranteed 40%, the analytics queue 30%, ad-hoc 30%, with sub-queues for teams within — so the organization's resource priorities are configuration, enforced automatically, rather than informal norms that break under load. The guarantee model is the key: a queue's capacity is assured when it needs it, so a team can count on its share regardless of cluster contention — the predictability that makes shared clusters viable for production workloads.
The elasticity-plus-preemption mechanism is what makes guarantees efficient rather than wasteful. A naive guarantee (each queue always gets exactly its share, idle or not) wastes resources: the finance queue's 40% sits idle overnight while analytics queues. Elasticity lets queues borrow idle capacity (analytics borrows finance's idle 40%, using 70% of the cluster overnight) — no waste. But borrowing creates a problem: when finance's jobs arrive in the morning, its guaranteed 40% is being used by analytics. Preemption solves it — the scheduler reclaims the over-borrowed capacity (killing or waiting for analytics containers) to restore finance's guarantee. The elasticity-preemption pair is the elegant core: guarantees are honored (preemption restores them) and idle capacity is shared (elasticity lends it) — the best of both, at the cost of preemption's disruption (killed containers redo work). Tuning this balance (how aggressively to preempt, how much to allow borrowing) is the scheduler's central operational art.
And the within-queue and hardware dimensions complete the multi-tenancy. User limits prevent one user from monopolizing a queue (the analytics queue's 30% shared fairly among analysts, not grabbed entirely by one). Node labels partition heterogeneous hardware (GPU nodes labeled and reserved for ML queues, so a non-ML job can't occupy scarce GPUs). Resource types schedule memory, vcores, and GPUs together (a job needing 4 GPUs and 64GB is scheduled against all dimensions). These make the scheduler handle real multi-tenant complexity — many users, heterogeneous hardware, multi-dimensional resources — rather than just splitting a homogeneous pool, which is what production shared clusters actually require.
The architecture: every piece explained
Top row: the capacity model. The queue hierarchy mirrors the organization: a root queue divided into child queues (finance, analytics, ml), each possibly sub-divided (finance → reporting, risk) — a tree where capacities sum to 100% at each level. Capacity guarantees: each queue has a configured capacity (its guaranteed minimum share — finance 40% means finance is assured 40% of the cluster when it needs it). Elasticity: a queue can use more than its guarantee by borrowing idle capacity from sibling queues — so guarantees don't waste idle resources; a queue with idle capacity lends it. Maximum capacity: a ceiling on borrowing (finance can borrow up to, say, 60% max) — bounding how much any queue can grab, preventing a borrowing queue from taking the whole cluster and making preemption too disruptive.
Middle row: reclamation and within-queue control. Preemption reclaims over-borrowed capacity: when a queue below its guarantee has pending demand and other queues are over their guarantee (borrowing), the scheduler preempts (reclaims containers from the borrowers) to restore the under-served queue's guarantee — the mechanism that makes guarantees real despite elasticity. User limits: within a queue, limits ensure fairness among users (a minimum user limit percent, so one user can't monopolize the queue). Priorities and ordering: applications within a queue are ordered (FIFO by default, or fair ordering) and can have priorities, determining which pending app gets resources first. Resource types: the scheduler allocates multiple resources (memory, vcores, and increasingly GPUs) together — a job's request is multi-dimensional, and scheduling considers all dimensions (dominant resource fairness).
Bottom rows: hardware and comparison. Labels and placement: node labels partition the cluster (GPU nodes labeled 'gpu', reserved for queues configured to access that partition) — so heterogeneous hardware is allocated deliberately (ML queues get GPU nodes, others don't). vs Fair Scheduler: the Capacity Scheduler emphasizes guarantees (each queue assured its capacity) while the Fair Scheduler emphasizes fair-share (resources divided fairly among running apps, converging to equal shares) — different philosophies (guarantee-oriented vs fairness-oriented), with the Capacity Scheduler favored for organizations wanting predictable per-team guarantees. The ops strip: queue sizing (capacities matching organizational priorities and actual demand — the political-technical core), starvation prevention (ensuring no queue or user is perpetually starved — user limits, preemption, ordering), and utilization monitoring (cluster utilization, per-queue usage vs guarantee, preemption rates — the health of the sharing).
End-to-end flow
Trace a day on a shared cluster with the Capacity Scheduler. Configuration: root split into finance (40%), analytics (35%), ml (25%), with ml's queue configured to access the GPU-labeled nodes. Overnight, finance's ETL is idle and analytics is running heavy reports: elasticity lets analytics borrow finance's idle 40%, using ~75% of the cluster (its 35% plus borrowed 40%) — no waste, the idle finance capacity productively used. Morning arrives: finance's ETL jobs submit, needing their guaranteed 40%, but analytics is using it. The scheduler preempts: it reclaims containers from analytics (which was over its guarantee, borrowing) to restore finance's 40% — analytics drops back toward its 35% guarantee, finance gets its assured share, the ETL runs on time. The elasticity-preemption pair delivered both efficiency (overnight sharing) and guarantee (morning reclamation) — the scheduler's core value.
The within-queue and hardware vignettes show the finer control. Within the analytics queue, three analysts submit jobs; user limits ensure they share the queue's capacity fairly (one analyst can't grab all 35% — the user limit caps each), so no analyst starves. The ML queue's jobs need GPUs; node labels ensure ML jobs are placed on the GPU-labeled nodes (and non-ML jobs can't occupy those scarce GPUs), so the expensive GPU hardware serves the workloads that need it. A multi-dimensional scheduling case: an ML job requests 4 GPUs and 128GB memory; the scheduler allocates against both dimensions on the GPU nodes, considering the dominant resource — the multi-resource scheduling real ML workloads require.
The tuning and operational vignettes complete it. The team faces a preemption-tuning decision: aggressive preemption restores guarantees fast (finance gets its share immediately) but disrupts analytics (killed containers redo work — wasted compute); gentle preemption is less disruptive but slower to honor guarantees. They tune to their priorities (production finance guarantees matter most, so reasonably aggressive preemption for finance, gentler elsewhere) and monitor preemption rates (high preemption signals mis-sized queues or over-aggressive borrowing). They also handle a starvation risk: a low-capacity queue with steady demand was occasionally starved when higher-capacity queues borrowed aggressively; adjusting max-capacity ceilings and preemption ensured the small queue got its guarantee. The consolidated discipline the team documents: size queues to organizational priorities and actual demand, tune elasticity (max capacities) and preemption to balance efficiency against guarantee-honoring and disruption, use user limits to prevent within-queue monopolization, use labels for heterogeneous hardware, and monitor utilization, per-queue usage, and preemption rates — because the Capacity Scheduler encodes the organization's resource priorities as enforced policy, and getting the queue sizing and elasticity-preemption balance right is what makes a shared cluster both efficient and predictable.