Every inference server has a front door, and its most consequential decision is not which admitted request to run next — it is which requests to refuse entry to at all. A GPU serving a large language model has a hard capacity: so many KV-cache blocks, so many concurrent sequences, so many tokens per second. Traffic does not respect that ceiling, and when arrivals exceed it the server has two options, queue or reject. A system that only knows how to queue turns an overload into an outage while its dashboards insist the GPUs are busy. Admission control decides what not to let in, so that what you did admit still lands inside its budget.
The front door, not the dispatch desk
Two scheduling decisions live in an LLM server and they are constantly confused. Admission control decides whether a request enters the system at all. Iteration-level scheduling — continuous batching — decides, among requests already inside, which occupy the batch on the next forward pass, which get preempted, and whose KV blocks get evicted. Our companion article on continuous batching owns that second problem entirely.
The boundary matters because the two have different powers. The batch scheduler can reorder and preempt, but it cannot create capacity; hand it more work than the GPU can finish and all it can do is spread the pain evenly. Only admission control can reduce the offered load. Its veto is the only thing standing between a traffic spike and a queue that swallows the service. Everything downstream is allocation; this is the gate.
Why the unbounded queue is the classic failure
Almost every server framework defaults to accepting the connection and appending the request to a queue bounded by nothing but memory. Under normal load this is invisible and free. Past saturation it is the single most destructive design in serving.
When arrival rate exceeds completion rate, throughput does not rise: the GPU was already saturated and completes the same tokens per second as before. The queue absorbs the entire surplus, so queue length grows linearly with time and so does waiting time; latency has no equilibrium to settle at. Then the second-order disaster arrives, because clients have timeouts. A request that waited past its deadline is abandoned by the caller but still sits in the server's queue, and when it reaches the GPU the server spends real capacity generating tokens nobody is listening to. At steady overload every completion is wasted work: goodput collapses toward zero while utilisation reads 100%.
Sizing the cap from the SLO, not from memory
The fix is a bounded queue, and the bound must come from your latency objective, not from how many request objects fit in RAM. Little's Law gives the conversion: for a stable system, average wait equals average queue length divided by completion rate. Invert it: if the server completes roughly 10 requests per second and your time-to-first-token budget allows 2 seconds of queueing, a queue deeper than about 20 requests is a promise you cannot keep.
That is the whole calculation, and it is why “we set the queue to 10,000” is never an answer — a queue that deep is a machine for manufacturing timeouts. Two refinements make it usable: measure the completion rate continuously, since it varies with prompt length and batch composition, and express the cap in the resource that actually binds — queued tokens or reserved KV blocks, because one request can be worth fifty.
Shedding load: the 429 contract
Once the cap is hit the server must reject, and a rejection is an interface with a contract. Return 429 Too Many Requests for a client that exceeded its own share, and 503 Service Unavailable when the fleet is out of capacity — that tells the caller whether to back off or fail over. Always include Retry-After, derived from measured drain time.
The overriding requirement is that rejection be cheap. If shedding costs as much as serving, you have built a second queue in front of the first: reject before tokenising, before allocating KV blocks, before touching the GPU. The failure mode to design against is the retry storm, since a thousand clients rejected at the same instant retry at the same instant unless their backoff is jittered. Count and alert on rejections too — a silently shedding server looks healthy from the inside.
The LLM wrinkle: cost is unknown at the door
Classical admission control assumes requests are roughly interchangeable, so counting them approximates counting work. Autoregressive generation breaks that assumption harder than almost any other workload: a request's cost is dominated by its output length, which is not known until generation stops. Two identical prompts can differ by two orders of magnitude in GPU time depending on whether the model emits a sentence or a thousand-line file.
The gate is therefore admitting a lottery ticket. Prefill cost is at least predictable — it scales with prompt length, superlinearly in attention — but decode cost, and the KV footprint that grows with every token, are genuinely unknown. This is why memory pressure arrives late: a batch that fits comfortably at admission can exhaust the KV pool thousands of steps later, forcing exactly the preemptions admission control was meant to prevent.
Estimating the cost anyway
Unknown does not mean unusable; the gate needs an estimate, and several signals exist before a single token is generated. Prompt length is exact. The client-supplied max_tokens is a hard upper bound. Route, model, and tenant carry strong historical priors — a summarisation endpoint and a code-generation endpoint have entirely different length distributions, and the per-route distribution beats any global average.
Two policies follow. Conservative admission reserves the worst case (max_tokens): it never overcommits, but wastes capacity because most requests finish early. Optimistic admission reserves a percentile of the historical distribution and accepts occasional preemption when it guesses low. Most systems sit in between, admitting against a high percentile and reconciling as real consumption becomes visible. Requiring max_tokens turns an unbounded unknown into a bounded one, which is worth the API friction.
Queue-time budgets and dropping the doomed
A bounded queue stops unbounded growth but not waste. A request that has already waited longer than its latency budget allows has, in the only sense that matters, already failed — running it consumes capacity that could have rescued a request still capable of succeeding. Attach a deadline at arrival and check it at dequeue: if the remaining budget is smaller than the estimated service time, drop it rather than starting work you know will miss.
This is deadline-aware admission, and it is what keeps goodput from collapsing during a spike. A counterintuitive companion: under sustained overload, serving the queue LIFO beats FIFO, because the newest request has the most budget left, whereas FIFO faithfully serves the oldest — and therefore most nearly expired — request first, so every completion is a near-miss. Propagate deadlines from the caller so the chain agrees on when a request stops being worth finishing.
Priority by tenant and by request class
Not all rejections are equally acceptable, so the gate should discriminate. The axes that matter are who is asking and what kind of work it is. Interactive chat needs a fast first token and is worth admitting ahead of a batch summarisation job measured in minutes; a paying tenant outranks free tier; a health check should never be shed by accident.
Implement this as reserved capacity rather than strict priority ordering. Give each class a guaranteed floor and a ceiling it cannot exceed, so one tenant's burst cannot consume the pool and lower classes are not starved indefinitely. Weighted fair queueing across tenants, with per-tenant token-bucket limits in front, survives a noisy neighbour. The rule: when the gate must shed, it sheds the lowest class first and the highest never — and the classes are declared in advance, not invented during the incident.
Backpressure that reaches the client
Shedding is only half the loop. Backpressure is the signal travelling back up the call chain telling upstream components to slow down, and it is what stops the gate from having to reject at all. It works only if every hop honours it. A load balancer that keeps routing to a saturated replica, or a gateway with its own unbounded buffer, absorbs the signal and reintroduces the queue you just removed one layer up.
Concretely: bound every queue at every tier, propagate 429 and 503 rather than retrying them locally, and let HTTP/2 or gRPC flow control apply connection-level pressure when a streaming consumer reads slowly. On the client side, adaptive concurrency — an AIMD-style controller that cuts in-flight requests on rejection and raises it slowly on success — lets callers converge on the server's real capacity instead of hammering it. A client that reduces its own send rate is worth more than any server-side cleverness.
Tuning it, and what to watch
Admission control has few knobs — queue depth cap, queue-time budget, per-class reservations, estimation percentile — and none can be chosen from first principles. They come from measurement: find the load at which latency turns the knee, set the caps just below it, and re-derive them whenever the model, hardware, or traffic mix changes. Our load-testing article covers finding that knee; the SLO article covers defining the targets.
Afterwards you watch a different dashboard from the usual utilisation one. Track admitted versus rejected counts split by class and reason; queue wait at p99, not the mean; the error between predicted and actual output length; and above all goodput — requests completed within their SLO, not requests completed. A system whose throughput is flat while its rejection rate rises is working as designed. One whose GPUs are pinned at 100% while goodput falls has no front door at all.