YARN Preemption

What preemption solves: queue starvation and fairness

Imagine two queues in a YARN cluster: engineering with a 50% capacity guarantee, and analytics with 30%. On a quiet afternoon, engineering submits a single batch job that sprawls across the entire cluster—300 containers, using 100% of available memory. An hour later, analytics submits a real-time query that needs 50 of its allocated nodes to meet its SLA. Without preemption, that query waits for the batch job to finish, even though analytics owns those resources by policy. With preemption, the scheduler interrupts the batch job, kills some of its containers, and hands the freed resources to analytics. The batch job restarts those containers later when analytics releases capacity back.

Starvation is the nightmare: if one queue is allowed to occupy resources indefinitely beyond its guarantee, others never run. Preemption enforces the minimum allocation promise that every queue's guaranteed resources are actually available, regardless of current demand elsewhere. It is not voluntary—jobs killed by preemption have no say in whether they want to yield.

The preemption problem: why not just wait?

A scheduler could avoid preemption by simply refusing to allocate resources to one queue if another is starved. That pushes the problem: the queue waiting for preemption sits idle, and jobs already running consume resources. The moment you allow jobs to accumulate pending tasks beyond the free pool, you create a queue. If that queue never clears—because a different job is occupying the cluster—the starved queue is stuck forever.

Preemption breaks the deadlock by asserting: this queue's guarantee is inviolable. If resources are needed elsewhere, they will be taken. The consequence is that running jobs must be robust to interruption. They do not crash—YARN leaves the JVM running and the TaskTracker / NodeManager re-runs the task—but the task loses its container and has to be rescheduled, which wastes compute and delays completion. So preemption is a policy trade: some inefficiency today to guarantee progress tomorrow.

How preemption works: the container kill mechanism

When the scheduler decides preemption is needed, it picks a set of containers to kill—typically those from the queue that is furthest above its fair or allocated minimum. It sends a STOP_CONTAINER request to the NodeManager hosting the container. The NodeManager does not kill the JVM; instead it signals the Task to stop gracefully. If the task does not die within a grace period (default 15 seconds), the NodeManager escalates to a hard kill.

The grace period is the opportunity: a well-behaved task uses those 15 seconds to save state, close files, and exit cleanly. The Application Master (AM), watching for container-lost events, re-runs the task on a new container. From the job's perspective, a preempted task is like a task that failed—it gets rescheduled. The key insight is that preemption does not lose data; it interrupts progress. A MapReduce job re-runs the killed tasks; a Spark job re-computes the RDD partition; a streaming job replays or re-processes the window. The cost is recomputation, not correctness.

Preemption in the Fair Scheduler

The Fair Scheduler enforces fair shares: each queue should get its allocated fraction of the cluster. If a queue falls below its fair share, the scheduler hunts for containers to kill. The Fair Scheduler's preemption strategy is relatively simple: find the queue furthest above its fair share and kill its containers until the starved queue is satisfied.

Fair preemption is configurable via preemptionInterval (how often to check for starvation) and minSharePreemptionTimeout (how long a queue must wait below its minShare before preemption triggers). The scheduler also respects fairSharePreemptionTimeout, which gates preemption based on fair-share guarantees. A queue can opt out of preemption entirely by setting allowPreemption=false, useful for critical jobs where interruption is unacceptable. This flexibility comes at a cost: that queue may starve others, so it is a deliberate trade-off, not a default.

Preemption in the Capacity Scheduler

The Capacity Scheduler layers queues into a hierarchy and assigns each a guaranteed capacity and maximum capacity. Preemption is an off-by-default toggle: set yarn.resourcemanager.scheduler.monitor.enable = true and name your CapacitySchedulerPreemptionPolicy to activate it. Once enabled, the Capacity Scheduler's preemption monitor runs at a configurable interval (default 3 seconds) and checks each queue's allocation against its guarantee.

Capacity preemption is hierarchical and graduated: a child queue below its guaranteed capacity asks its parent, which asks its siblings. The scheduler builds a preemption list—which containers from which queues are candidates for killing—and applies it gradually: it might wait one cycle before actually killing, giving containers a chance to finish naturally. This staged approach reduces thrashing. The administrator controls yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round to limit how many containers are killed per monitoring cycle, preventing a avalanche of interruptions.

The grace period: giving tasks time to exit cleanly

When the scheduler marks a container for preemption, the NodeManager does not immediately kill it. Instead, it signals the running task with a SIGTERM or equivalent, then waits. The grace period—configurable via yarn.resourcemanager.scheduler.monitor.grace-period.ms (default 15000 ms)—gives the task time to shut down gracefully. A well-written task catches the signal, flushes buffers, saves intermediate state, and exits.

The catch is that not all tasks cooperate. A Java process may ignore SIGTERM and keep running; a shell script may ignore the signal entirely. After the grace period expires, the NodeManager escalates to SIGKILL, which cannot be caught. For MapReduce, this is mostly transparent—tasks are stateless, and re-running them is routine. For a Spark streaming task holding connection state or a custom application with in-process state, an ungraceful kill can cause data loss or corruption. Increasing the grace period gives more time but delays preemption; decreasing it speeds preemption but risks ungraceful exits.

Configuration and tuning preemption

Fair Scheduler preemption is controlled by the queue config, typically in fair-scheduler.xml. Set <minSharePreemptionTimeout>60</minSharePreemptionTimeout> (in seconds) to wait one minute before preempting for minShare; <fairSharePreemptionTimeout>30</fairSharePreemptionTimeout> for fair-share preemption. Disable preemption for critical queues with <allowPreemption>false</allowPreemption>.

Capacity Scheduler tuning involves capacity-scheduler.xml: set yarn.resourcemanager.scheduler.monitor.enable=true, choose an interval like yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval=3000 (ms), and cap aggressive preemption with yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round=0.1 (10% of cluster per cycle). Increase yarn.resourcemanager.scheduler.monitor.grace-period.ms to 30000 if you need more time for graceful shutdown. Start conservative and tune up: aggressive preemption that kills hundreds of containers per minute can destabilize long-running jobs and analytics queries.

Monitoring and detecting preemption

Preemption is invisible if you do not look for it. The YARN ResourceManager logs are the primary signal: look for CONTAINER_EXPIRED or preemption in the logs. The ResourceManager web UI shows queue allocations in real time, and a queue's allocation oscillating around its guaranteed capacity often signals active preemption. More directly, check the Application Master logs for Container [ID] is running beyond physical memory limits or Container preempted messages.

For quantitative signals, export QueueMetrics from the ResourceManager. The metrics PresourcesPreempted (containers killed), Preemptions (number of preemption events), and PreemptedMemoryMB/PreemptedVcores (resources reclaimed) paint a picture of preemption frequency and intensity. A cluster with high preemption rates—dozens of preemptions per hour—suggests either aggressive scheduling policies or contention that justifies those policies. Baseline these metrics over a week to separate normal from anomalous.

Impact on job performance and latency

Preemption is expensive. A job killed mid-task loses all progress in that task—a MapReduce job re-runs the mapper or reducer from the beginning, a Spark job re-computes an RDD partition. The re-run consumes compute again, and the task is delayed by the time to kill, wait for scheduler to re-allocate, and re-run. For short tasks (seconds), a preemption might add 5-10 seconds of wall-clock time. For long tasks (minutes), preemption stalls the entire job.

The severity depends on task characteristics. Short-lived batch jobs tolerate preemption well: the re-run is a small fraction of total time. Long-running streaming or interactive jobs suffer more: every preemption event stalls query latency or breaks a stream window, visible to end users. Stateful applications (those storing in-process state like connection pools or caches) can lose data if preempted ungracefully. Design job applications to be preemption-aware: checkpoint state, keep tasks short, and gracefully handle SIGTERM.

Trade-offs: fairness vs stability

Preemption is a fairness mechanism, not a stability mechanism. Enabling preemption guarantees that every queue's minShare will be available, but it introduces a new class of failure: preemption-induced delays and jitter. A job that could complete in 10 minutes may take 15 if containers are killed and re-run mid-flight. For batch workloads (ETLs, report generation), this is tolerable—late is late. For latency-sensitive work (real-time analytics, serving backends), preemption can violate SLAs.

The trade-off is explicit: fair clusters are noisier. You choose whether you want guaranteed queue access (preemption on) or predictable job latency (preemption off). In practice, most shared clusters use preemption sparingly: enable it only when queue starvation is empirically a problem, not by default. For critical latency-sensitive queues, allocate generous capacity buffer (e.g., 40% instead of 30%) so starvation is unlikely and preemption rare. Use allowPreemption=false on critical queues if your scheduler supports it, accepting that they may be slightly unfair in return for stability.

Common gotchas and pitfalls

Gotcha 1: Preemption happens before you think. Many admins enable preemption but forget to tune timeouts, so the first queue that overshoots by even 1% triggers a cascade. Start with long preemption timeouts (300+ seconds) and tighten only if starvation is empirically observed.

Gotcha 2: Tasks ignore SIGTERM. Java processes often do not handle SIGTERM gracefully; they need ShutdownHook registration or custom signal handlers. Without this, the grace period expires and the JVM is killed hard, losing state and buffers. Audit your applications for signal handling before enabling preemption.

Gotcha 3: Preemption thrashing. If preemption monitor intervals are too short or thresholds too tight, the cluster can enter a state where containers are killed faster than applications can reschedule them, causing CPU and memory spikes as tasks constantly re-run. Limit preemption rate with total_preemption_per_round and increase monitor intervals to 5+ seconds.

Gotcha 4: Not seeing preemption. Preemption is often silent in metrics if you do not export them. A queue appears to be oscillating rather than being preempted. Always instrument container loss events and attribute them to preemption, starvation, or node failure.

Best practices for production preemption

1. Start with preemption disabled. Baseline your cluster without it. Identify actual starvation events via metrics. Only enable preemption if starvation is a recurring, user-facing problem, not a theoretical concern.

2. Tune preemption for batches, not interactive jobs. If your cluster runs mostly ETL and reporting (batch), preemption is a good match. If it serves low-latency queries or real-time analytics, preemption creates jitter; instead, size queues with more breathing room and rely on autoscaling or careful reservation.

3. Increase grace periods and monitor intervals. Use at least 30 seconds for grace period and 5+ seconds for monitor interval. Aggressive preemption (15s grace, 1s monitor) creates high churn; conservative tuning (60s grace, 10s monitor) gives jobs time to finish naturally and preemption rarely triggers.

4. Protect critical queues. Set allowPreemption=false for production services or SLA-critical workloads. Accept slight unfairness for stability.

5. Monitor preemption rate and impact. Export metrics on containers killed, preemption frequency, and task re-run counts. Alert on sudden spikes. Correlate job latency with preemption events to quantify the impact.

Preemption versus other resource reclamation strategies

YARN preemption is aggressive: it interrupts running tasks. Other strategies exist. Reservation systems (Hadoop Reservation System, Kubernetes PodDisruptionBudgets) let workloads opt into preemption—they declare ‘this job can tolerate up to N simultaneous preemptions’ and the scheduler respects it. This is gentler but requires application buy-in.

Demand-driven allocation (yielding idle capacity to queues without preemption) avoids interruption entirely but requires strict admission control—once a queue gets capacity, it must give it back. This works well for batch workloads with clear start/end points but fails for long-running services.

Node draining (gracefully evicting all workloads from a node before maintenance) is preemption's gentler cousin: it gives containers time to drain, but it still disrupts. Dynamic quotas (adjusting queue limits in response to demand) avoid preemption by preventing overallocation in the first place. The best production clusters often combine strategies: preemption for batch, reservations for interactive, node-draining for maintenance.

When to enable preemption: signals and indicators

Enable preemption if you observe: (1) Persistent queue starvation—a queue is consistently below its guaranteed minimum for hours, with large pending queues; (2) Greedy single jobs—a batch job regularly consumes 100% of cluster capacity, blocking other queues; (3) User complaints about long waits—queue owners report that their jobs do not start for hours despite having reserved capacity; (4) Cluster utilization far below capacity—you have room to grow but some jobs block others from launching.

Do not enable preemption if: (1) Cluster is mostly empty—no contention, so starvation is not happening; (2) Interactive/real-time workloads dominate—the added latency from preemption violations your SLAs; (3) Jobs are long and stateful—preemption is disruptive and re-run costs are high; (4) Queue guarantees are generous—if queues rarely overallocate because they are sized with buffer, preemption is unnecessary. In most environments, preemption is a rare tool, not a default setting.