Why architecture matters here
The architecture matters because GPU utilization is the entire economics of LLM serving, and static batching squanders it. A language model's decode step is memory-bandwidth bound: the cost is dominated by reading the model's weights from GPU memory, and that read happens whether the batch has one sequence or fifty. So the marginal cost of adding a sequence to a decode step is nearly free until the batch is large — which means throughput scales almost linearly with batch size until saturation. Static batching, by letting slots go idle as short requests finish, runs at a fraction of the batch size it paid for. Continuous batching keeps the batch full, extracting the linear throughput the hardware makes available.
The variance in output length is what makes this dramatic rather than marginal. In real traffic, response lengths span two orders of magnitude — a yes/no answer next to a long generation — so in a static batch the longest sequence dictates how long every slot is held. With a mix of lengths, a static batch can waste more than half its compute waiting on tail sequences. Because iteration-level scheduling releases each slot the moment its sequence ends and refills it immediately, the wasted time collapses to at most one iteration, and effective utilization approaches the ideal. The higher the length variance, the bigger the win.
It matters, second, because it decouples the two phases of inference that have opposite hardware profiles. Prefill — processing the prompt to populate the KV cache — is compute-bound and processes many tokens in parallel; decode — generating one token at a time — is memory-bound. A good continuous-batching scheduler interleaves them, using prefill to soak up compute headroom while decode saturates memory bandwidth, so neither resource sits idle. The scheduler's admission policy — how aggressively it injects prefill of new requests into a batch of decoding ones — is one of the main levers on the throughput/latency balance.
Finally, it matters because KV-cache memory, not compute, is usually the real capacity ceiling, and continuous batching only pays off if that memory is managed well. Every active sequence's KV cache occupies GPU memory proportional to its length, and the sum across the running batch can easily exceed what the GPU has. Naive contiguous allocation of a max-length cache per sequence wastes enormous memory to internal fragmentation and caps the batch far below what compute could handle. Paged attention removes that waste by allocating small blocks on demand, so the batch can grow until memory is genuinely full rather than until the fragmentation cliff. The scheduler and the pager are thus two halves of one machine: iteration-level scheduling keeps the batch full in time, and paged memory keeps it full in space, and only together do they deliver the throughput that makes an expensive accelerator economical — which is why every serious inference server (vLLM, TensorRT-LLM, TGI) is built around this pair.
The architecture: every piece explained
The scheduler is the heart of continuous batching. Each iteration it decides which sequences run: it keeps the currently-decoding sequences, removes any that finished last step, and admits waiting requests from the queue into the freed capacity — subject to compute and KV-memory limits. It runs one forward pass over the assembled batch, samples one token per sequence, and repeats. The admission and eviction happen between iterations, which is what 'iteration-level scheduling' means and what lets batch membership change continuously.
The running batch is a heterogeneous set: some sequences are being prefilled (their prompt processed in one big step), others are decoding (one token). Modern schedulers mix these — chunked prefill splits a long prompt's prefill across iterations so it does not stall ongoing decodes — so a new request's prompt processing is interleaved with in-flight generation rather than blocking it. This mixing is what keeps both compute and bandwidth busy.
The paged KV cache stores each sequence's attention state in fixed-size blocks. Rather than reserving a contiguous max-length buffer per sequence, the pager hands out blocks as a sequence grows and reclaims them when it finishes. A per-sequence block table maps logical token positions to physical blocks — exactly like virtual memory — so caches need not be contiguous and the memory of a finished sequence is instantly reusable by a new one. This is what makes dynamic batch membership feasible without fragmentation.
The block allocator manages the pool of KV blocks: allocating on demand, freeing on completion, and — when memory runs short — driving preemption. Preemption is the pressure-relief valve: when admitting or growing sequences would exceed KV memory, the scheduler evicts one or more running sequences, either recomputing their KV later (drop and redo prefill) or swapping their blocks to CPU memory and back. The forward step is the actual GPU computation — one transformer pass over the batch producing next-token logits — and token sampling and streaming turn those logits into emitted tokens sent back per sequence as they are produced. Together these pieces — the iteration-level scheduler, the mixed prefill/decode batch, the paged KV cache and its block allocator, preemption under memory pressure, and per-sequence streaming — are the complete continuous-batching engine, and each addresses a specific obstacle to keeping the GPU fully and continuously utilized.
End-to-end flow
Trace a few iterations of a busy server. At iteration t the running batch holds eight decoding sequences, each generating its next token, plus one new request whose prompt is being prefilled (in a chunk, so it shares the step without stalling the decoders). The scheduler runs one forward pass over all nine, samples a token for each decoding sequence, and advances the prefill. Every active sequence made exactly one step of progress; the GPU processed a full batch.
At iteration t+1, one of the eight decoders emitted its end-of-sequence token — its answer is complete. The scheduler removes it from the batch and the block allocator frees all of its KV blocks back to the pool. That freed capacity — both a batch slot and KV memory — is immediately available. The scheduler pulls the next request off the queue and begins its prefill in the same iteration, so the batch is back to full occupancy. The finished sequence's departure cost the GPU nothing; its slot was refilled the very next step.
Now a memory-pressure moment. Several long sequences have been decoding for hundreds of steps, and their KV caches have grown until the block pool is nearly exhausted. A new admission would need blocks that are not available. The scheduler invokes preemption: it selects a sequence (by policy — often the most recently admitted, to preserve fairness for older requests), frees its KV blocks — either swapping them to CPU memory or discarding them to be recomputed later — and uses the reclaimed space to keep the rest of the batch running. The preempted sequence is returned to the queue and resumed once memory frees up, its KV either swapped back or rebuilt by re-running prefill. From the client's view its stream pauses briefly rather than failing.
Meanwhile each token, as it is sampled, is streamed back to its own client — the server does not wait for a sequence to finish to return its output. The net effect over many iterations is a GPU that runs a full batch on essentially every step: sequences flow in as capacity appears and out as they complete, prefill and decode interleave to keep both compute and bandwidth busy, and paged memory lets the batch grow until memory is truly full. The throughput the operator paid for — near-linear in batch size up to the memory ceiling — is actually realized, because the two idle-time sources of static serving (slots waiting on tail sequences, and memory lost to fragmentation) have both been designed away. That continuous recomposition of the batch, iteration by iteration, is the whole architecture, and it is why the same GPU can serve several times the requests per second it would under static batching.