Preemption is the decision to take a GPU away from work that is already running on it. It sounds like one problem and it is at least three, separated by the unit you reclaim and the clock you have to do it on: a single sequence inside an inference server, measured in milliseconds; a training job on a shared cluster, measured in minutes; and a whole machine under a cloud reclamation notice, measured in whatever seconds the provider gives you. Each needs different machinery, because the state you must preserve lives somewhere different and costs something different to move. What unifies them is an unfriendly fact: the GPU itself will not help you. Its preemption support switches which context executes, but it does not hand back the resource you are actually fighting over.
Three regimes, three clocks
Sort preemption by two questions and the confusion clears: what is the unit you take back, and where does its state live? At request level the unit is one sequence in a decode batch, its state is KV cache in HBM, and the deadline is the gap between two decode steps. At job level the unit is a distributed training run, its state is parameters plus optimizer moments plus data-loader position spread over many ranks, and the deadline is however long the scheduler will wait after it asks. At instance level the unit is the whole node and the deadline is set by a provider you do not negotiate with.
Those three clocks are why you cannot build one mechanism. A millisecond budget rules out anything touching durable storage. A fixed notice window rules out anything whose cost scales with model size unless you have already paid most of it in advance.
What the hardware gives you, and why it is not enough
Modern datacenter GPUs support fine-grained preemption of a running context: the driver can stop a resident kernel, save the execution state of its warps, and let another context run, rather than waiting for every thread block to retire. Its mechanics — work distributor, warp scheduling, MPS, time-slicing — belong to the GPU scheduling article, not this one.
The reason it does not solve your problem is one sentence: a context switch preserves the allocation, it does not release it. The preempted context's device memory stays exactly where it was, because that is the whole point of being able to resume. But in practical GPU workloads the scarce resource is HBM capacity, not SM time. Reclaiming memory means someone above the driver decides what to evict, moves it or throws it away, and knows how to rebuild it. That is application logic, and no hardware feature makes the decision for you.
Request level: the trigger and the unit
Inside a continuous-batching inference server, preemption fires when the block allocator cannot satisfy the next decode step. Every running sequence appends a token per iteration, so memory demand grows monotonically during decode while admission decisions were made earlier against a smaller footprint. The server holds a reserve of free blocks and, when the pool drops below it, stops the step and evicts somebody. Overcommitting on admission and correcting by preemption is deliberate: it lets you run near the memory ceiling instead of provisioning for every request's worst-case output length.
The unit you reclaim is one sequence's KV blocks, and the state you must preserve is tiny: the token ids generated so far, the sampling parameters, and the request's identity. The KV tensors are derived state — a deterministic function of those tokens and the weights. That property is what makes request-level preemption far cheaper than it looks.
Choosing a victim, and not starving it
Victim selection is a policy, and the common default is last-in-first-out: evict the most recently admitted sequence. That is not arbitrary. LIFO protects work that is nearly finished, so preemption discards the least accumulated compute, and it keeps rough arrival-order fairness because the newest arrival caused the pressure. Evicting the largest sequence frees the most memory per eviction and stops a cascade fastest, at the cost of punishing long-context users specifically. Evicting by priority tier is what you want when tenants have different contracts; evicting by SLO slack is the most principled and the hardest to estimate.
Whatever the policy, guard against livelock. A sequence preempted, resumed, and preempted again before it makes progress burns compute and produces nothing. The standard defences are an ageing term that makes a repeat victim progressively harder to re-select, and a hard cap on preemptions per request after which it is failed outright rather than churned forever.
Swap, recompute, and who pays the pause
Two ways to free the blocks. Swap copies the victim's KV to pinned host memory and back on resume, costing two transfers over a host link. Recompute frees the blocks immediately and re-runs prefill from the stored token ids when the sequence is rescheduled, costing one prefill. Recompute usually wins for moderate prompts because prefill is compute-bound and highly parallel while swapping is a bandwidth transaction on a link an order of magnitude slower than HBM; swap wins for very long contexts and for forked groups, where recompute repeats the same prefix work per branch.
The point most teams miss is who pays. The decode step is synchronous across the batch, so the eviction stall and the resume cost land on every concurrent request, not just the victim: a preemption surfaces as an inter-token latency spike for users who did nothing wrong. Preemption rate belongs on your latency dashboard, not only your memory one.
Draining an in-flight decode batch versus killing it
Rolling deployments and scale-downs are preemption too, and here you have a choice the memory-pressure case does not give you. A drain pulls the replica out of the load balancer, stops admitting new requests, and lets the in-flight batch decode to completion. Its cost is set entirely by the longest generation still running: one request with a large max-tokens budget holds a pod open for minutes while it serves a shrinking, increasingly inefficient batch.
So bound it. Set the termination grace period from your p99 generation length rather than a default, and pair the preStop hook that begins the drain with a hard deadline after which remaining sequences are failed with a retryable status or truncated with an explicit finish reason. State that contract for streaming clients: a truncated stream must be distinguishable from a completed one, or callers silently accept half an answer. Note too that Kubernetes Pod Disruption Budgets cover only voluntary disruptions — they do nothing about a provider reclaiming the node underneath you.
Job level: what a checkpoint actually contains
On a shared training cluster, preemption means a scheduler wants your GPUs for higher-priority work, and the only way to give them back without losing days is checkpoint-and-restore. A correct checkpoint is much more than weights: it holds the parameters, the optimizer state, the learning-rate schedule position, the RNG states for every rank, and the data-loader position — because a resume that reshuffles the data or restarts the schedule is a silently different training run, not a continuation.
Optimizer state dominates the size. A mixed-precision Adam job carries an fp32 master copy plus two fp32 moments; as an illustrative figure, budget on the order of 12–16 bytes per parameter rather than the 2 bytes a bf16 weight suggests. Sharding it lets each rank write its own slice, parallelising the write but making restore sensitive to a different world size.
Checkpoint cost sets the minimum useful preemption interval
Here is the constraint that governs the job-level regime. Writing a checkpoint costs wall-clock time C; restoring and re-forming the process group costs R; and if you checkpoint every I seconds, an unexpected preemption throws away I/2 seconds of work on average. Classical checkpoint theory puts the optimal interval near the square root of twice the checkpoint cost times the mean time between failures — the useful reading being that I must be much larger than C, or the cluster writes state instead of training.
Turn that around and it is a scheduler constraint: preempting more often than roughly C + R plus the lost work makes the job a net consumer of GPU-hours. If a checkpoint takes minutes, a scheduler that preempts every few minutes never lets it finish. This is why asynchronous, sharded checkpointing matters — staging state to host memory in seconds and flushing to durable storage in the background collapses C from the job's perspective, legitimising a much shorter preemption interval.
Spot reclamation: designing inside the notice window
Preemptible and spot instances give a termination notice — typically tens of seconds to a couple of minutes depending on the provider, delivered as an instance-metadata signal and usually surfaced to the pod as SIGTERM. Be honest about what fits. Writing a full sharded checkpoint to object storage does not. What fits is flushing state you have already staged: if an async checkpointer keeps a recent copy in pinned host memory, the handler only pushes bytes already off the GPU, and can prioritise the newest shard over completeness.
The second-order problem is worse. A collective-communication job is gang-scheduled: losing one rank hangs every other rank until a watchdog times out, so one reclaimed node stalls the entire run. The mitigations are structural — elastic launchers that detect the loss and re-rendezvous at a smaller world size, a warm on-demand pool to backfill critical ranks, and spreading a job across instance pools so one capacity event cannot take the whole run. Many teams settle on a mixed fleet: on-demand for ranks that must not move, spot for elastic capacity around them.
The bill: what preemption costs and how to watch it
Preemption never creates throughput. Every preemption destroys work — a prefill to redo, a transfer to pay, minutes of training to repeat — and you accept that bill to buy something else: bounded tail latency for high-priority requests, fairness between tenants, or a cluster that runs near full utilisation instead of holding capacity idle for arrivals that may never come. The tuning question is therefore never how to preempt faster; it is how rarely you can preempt and still meet the guarantee you sold.
Instrument accordingly. Track preemption rate and preemptions per request — a rising per-request figure is churn, not load. Track repeated work as a fraction of total compute, the direct cost of the policy. Track time-to-resume separately from time-to-preempt, because the resume path is where swap decisions and re-rendezvous logic actually get tested. And read a sustained high preemption rate as the symptom it is: at request level admission control is letting in more than the memory pool can carry; at job level the queue is oversubscribed. Preemption is a safety valve, and a valve that is always open means the pressure needs fixing upstream.