Why architecture matters here
The reason priority scheduling deserves a real architecture is that LLM serving has a uniquely awkward cost structure: a request is not a single unit of work but a long sequence of decoding steps, each producing one token, and while it runs it holds a growing block of KV-cache memory hostage. A single long-context batch request can occupy scarce GPU memory for many seconds. If the scheduler can only make decisions at request boundaries, an interactive request that arrives one step after a batch flood must wait for those batch requests to finish generating — potentially thousands of tokens later. The latency-sensitive traffic is held hostage by the throughput traffic, and no amount of extra capacity fixes it because the coupling is temporal, not volumetric.
Priority belongs in the serving layer, not the client, for the same reason rate limits do: trust and global knowledge. Clients cannot see the queue depth, the KV-cache pressure, or what other tenants are doing, so a client-declared 'urgent' flag is meaningless unless something authoritative arbitrates it. The scheduler is the one component that sits at the contended resource — the GPU pool — with a global view of every request in flight, so it is the only place where a decision like 'pause this batch request to admit that interactive one' can be made correctly and enforced.
The two objectives the scheduler balances are fundamentally in tension, and naming them clarifies the whole design. Latency is about time-to-first-token and time-per-output-token for interactive requests: humans notice hundreds of milliseconds, and a product SLA might demand first token under 500ms at p99. Throughput is about tokens per second aggregated across the cluster: batch jobs care only that the GPUs stay saturated so the total job finishes cheaply. A scheduler that optimizes purely for latency runs the GPUs half-empty waiting for interactive requests; one that optimizes purely for throughput packs the batch so tightly that interactive latency collapses. The art is serving both by giving interactive traffic scheduling priority while letting batch traffic backfill every otherwise-idle slot.
There is also a business-policy dimension that pure performance framing misses. Different requests are worth different amounts: a paying customer's live query is worth more than a free-tier experiment, and a revenue-critical checkout-assistant call is worth more than an internal analytics backfill. Priority classes are how that business judgment is encoded into the runtime, so that when the cluster is contended — which is exactly when the decision matters — the valuable work wins by explicit policy rather than by the accident of who submitted first. This turns scheduling from a purely technical optimization into the enforcement point for the product's economics.
Finally, priority scheduling is what makes a shared cluster viable at all. The alternative to sharing is static partitioning — dedicate GPUs to interactive, dedicate others to batch — which wastes capacity because the interactive partition sits idle overnight and the batch partition sits idle during a traffic spike. A priority scheduler over a single pool captures that idle capacity: batch backfills the interactive GPUs when human traffic is light, and interactive preempts batch when a spike arrives. The scheduler is the mechanism that lets one pool do the work of two while honoring the latency contract of the first, and that consolidation is usually the difference between a profitable inference product and one whose GPU bill sinks it.
It helps to see why the problem cannot be pushed to a load balancer in front of the cluster. A load balancer routes whole requests to servers, but the contention that hurts interactive latency happens inside a server, at the granularity of individual decoding steps against a shared KV-cache. Two requests on the same GPU compete step by step; only the component that controls the per-step batch — the scheduler — can arbitrate that. Routing can spread load, but it cannot decide that request B should yield a decode slot to request A this step and reclaim it the next, which is precisely the decision that keeps a chat responsive while a batch job runs on the same hardware.
The architecture: every piece explained
Follow the boxes. The top row is admission and dispatch. Incoming requests carry metadata: a priority class (interactive, default, batch), an optional deadline or SLA, a tenant identity, and token limits. Admission control is the gate: it classifies each request, applies per-tenant rate limits and concurrency caps, and rejects or defers work that would blow the cluster's memory budget before it ever reaches a queue — better to reject fast than to admit a request that will starve everything. Admitted requests land in priority queues, one logical queue per class, so the scheduler can always see the highest-priority waiting request in O(1). The scheduler is the brain: at each step it decides which waiting requests to admit into the running batch and which running requests, if any, to preempt.
The middle row is execution. The running batch is the set of requests currently being decoded together — continuous batching means this set is re-formed every step, with finished requests leaving and newly admitted ones joining. Preemption is the lever that makes priority real: when a high-priority request needs memory or a slot that a low-priority running request holds, the scheduler pauses the low-priority one, freeing its resources, and resumes it later. The GPU workers execute the actual prefill (processing the prompt) and decode (generating tokens) steps for the running batch.
The bottom row is memory and fairness — the two constraints that make LLM scheduling harder than CPU scheduling. The KV-cache pool is the scarce resource: every running request holds a block of KV-cache proportional to its context length, and the pool is finite, so admitting a request is really about reserving KV-cache blocks, and preempting one is about evicting or swapping them. There are two flavours of preemption: recompute, which drops the request's KV-cache and regenerates it from the prompt on resume (cheap memory, expensive compute), and swap, which copies the KV-cache to host memory and restores it later (expensive bandwidth, no recompute). The scheduler chooses based on which resource is tighter.
The fairness accountant is the safeguard against priority becoming tyranny. Priority alone is dangerous: if interactive always beats batch, a flood of interactive requests starves batch entirely, and if one tenant's interactive traffic dominates, other tenants in the same class are starved. The accountant tracks per-tenant consumption — tokens served, GPU-seconds used — over a window and bounds each tenant's share, so priority governs ordering while fairness governs totals. The two together produce the intended behaviour: important work goes first, but no single source of important work consumes the whole cluster.
The telemetry strip is not decoration; it is what makes the whole system tunable. The scheduler emits, per priority class, the queue wait time, the time-to-first-token, the sustained tokens-per-second, and the preemption count. Those signals are how an operator knows whether the interactive class is meeting its SLA, whether batch is being starved to the point of never finishing, and whether preemption is happening so often that the cluster is thrashing — spending more time swapping KV-cache in and out than actually decoding. A priority scheduler you cannot observe per class is a priority scheduler you cannot trust, because the whole point is differentiated service and only per-class metrics reveal whether the differentiation is actually happening.
End-to-end flow
Walk a contended moment through the system. Overnight, a batch job submits 5,000 long-context summarization requests. Admission control accepts them within the batch tenant's concurrency cap; they fill the priority queue's batch class and, since the GPUs are otherwise idle, the scheduler admits as many as the KV-cache pool holds into the running batch. The GPUs run at high utilization, tokens-per-second is excellent, and the batch job is making steady progress — exactly the throughput regime batch wants.
Now a human opens a chat and sends a message. The interactive request arrives, admission control tags it interactive, and it lands at the head of the interactive queue. The scheduler sees a waiting interactive request but the KV-cache pool is full of running batch requests. Because interactive outranks batch, the scheduler selects one or more batch requests to preempt: it swaps their KV-cache to host memory (or drops it for later recompute), freeing enough blocks to admit the interactive request immediately. Prefill runs on the interactive prompt, the first token streams back in a few hundred milliseconds, and the SLA is met — even though the cluster was fully loaded when the request arrived.
As the interactive request decodes, the scheduler backfills. The preempted batch requests are not abandoned; whenever a slot and KV-cache free up — because the interactive request finished, or because there is headroom — the scheduler resumes them from their swapped or recomputed state. From the batch job's perspective, its throughput dipped briefly while it yielded to the human and then recovered; from the human's perspective, the response was instant. The GPUs never went idle, and both classes got what they care about.
Consider the fairness path. Suppose a second tenant floods the interactive class with a thousand requests. Priority alone would let that tenant monopolize the whole cluster and starve the first tenant's interactive traffic. The fairness accountant intervenes: it sees the flooding tenant has consumed far more than its share of GPU-seconds in the window and deprioritizes its queued requests relative to the starved tenant, so the two interactive tenants converge toward their fair split. Priority still beats batch, but within a class no tenant runs away with the cluster.
Finally, the overload path — the case that separates a robust scheduler from a fragile one. Interactive traffic spikes past what the pool can serve within SLA even after preempting all batch. The scheduler cannot conjure GPUs, so it must degrade deliberately: admission control begins shedding the lowest-value interactive requests (or returning a fast 'try again' rather than queuing them behind a wait that would blow their deadline anyway), the fairness accountant keeps the remaining capacity split evenly, and batch is fully paused. The key property is that the system sheds load at the edge, quickly and by policy, rather than admitting everything and letting every request miss its deadline together. Controlled rejection of some requests beats uniform failure of all of them, and telemetry makes the shedding visible so operators can add capacity or tighten limits before the next spike.