Why architecture matters here
The architecture matters because the cost structure of LLM serving is dominated by an expensive, underutilized accelerator, and batching is the primary lever on that utilization. A modern inference GPU is memory-bandwidth-bound during decode: each step must read the entire model's weights to produce a token, and reading those weights to serve one sequence costs almost the same as reading them to serve fifty. Batching amortizes that fixed weight-read cost across many sequences, so the per-token cost falls steeply as the batch grows. Anything that keeps the effective batch small throws that amortization away.
The second reason is that generation length is unpredictable and highly variable, which is precisely what static batching cannot tolerate. You do not know when you admit a request how many tokens it will produce — it depends on the content the model generates. So a static batch inevitably mixes requests that will finish in three steps with ones that will run for a thousand, and there is no way to size or schedule the batch to avoid the long tail dragging the whole group. The variability is intrinsic to the workload, so the batching strategy must handle heterogeneity as a first-class concern, not an edge case.
The third reason is the two-phase shape of a request. Every request begins with a compute-heavy prefill — processing the entire prompt at once to populate the KV cache and emit the first token — and then enters a long memory-bound decode phase of one token per step. These phases have completely different performance profiles, and a scheduler that treats them uniformly wastes the GPU. Continuous batching's iteration-level control is what lets the system interleave a new request's prefill with the ongoing decode of others, keeping both the compute and bandwidth resources busy.
The fourth reason is that latency has two distinct components that trade off against throughput, and only fine-grained scheduling can balance them. Time-to-first-token (TTFT) is how long a user waits before generation starts — dominated by queueing and prefill. Time-per-output-token (TPOT) is how fast tokens then stream — dominated by decode-batch efficiency. Packing the batch fuller improves throughput and TPOT amortization but can delay a new request's prefill, hurting TTFT. The scheduler sits exactly at this tradeoff, and understanding it is how you tune for a chat product (TTFT-sensitive) versus a bulk pipeline (throughput-sensitive).
Finally, the architecture matters because it is what makes memory the binding constraint rather than compute, and memory management is where continuous batching lives or dies. Every active sequence holds a KV cache proportional to its length, and the sum of those caches is what actually caps the batch size — not the GPU's FLOPs. Continuous batching only delivers its throughput if KV memory is managed tightly enough to keep many sequences resident at once, which is why it is inseparable from paged attention and from a preemption policy for when memory runs short. The scheduler and the memory manager are two halves of one design.
The architecture: every piece explained
The heart of the system is the iteration-level scheduler. Unlike a static batcher that makes one admission decision per batch, this scheduler makes an admission and eviction decision every decode step. On each iteration it looks at the currently running sequences, the pool of free KV memory, and the waiting queue, and decides the composition of the batch for the step about to execute: which sequences continue, which finished ones to retire, and which queued requests to admit into freed capacity.
A request in the running batch is a sequence with associated state: its generated tokens so far and, critically, its KV cache — the keys and values for every token it has processed, which the attention mechanism must read on every subsequent step. This cache grows by one token's worth of state each decode step and is the dominant consumer of GPU memory. When a sequence finishes, freeing its KV cache is what creates room to admit a new request.
Prefill and decode are the two operations the scheduler interleaves. When a new request is admitted, it must first be prefilled: the whole prompt is run through the model in one compute-intensive pass to build its initial KV cache and produce the first token. Thereafter it is in decode, contributing one token per step alongside the others. A well-designed scheduler can slot a prefill in between decode steps (or chunk a long prefill across several steps) so that admitting a new request does not stall the ongoing generation of the batch — this interleaving is what keeps both the compute-bound and bandwidth-bound resources productive.
Paged attention is the memory-management layer that makes all of this feasible. Rather than reserving one large contiguous KV region per sequence sized for its maximum possible length — which would waste enormous memory on sequences that finish early and fragment the rest — paged attention allocates the KV cache in fixed-size blocks, like virtual-memory pages. A sequence grabs blocks as it grows and returns them when it finishes; free blocks go back to a shared pool any sequence can draw from. This near-eliminates fragmentation and lets the scheduler pack far more concurrent sequences into the same memory, directly raising the achievable batch size.
The final piece is the preemption / eviction policy for when demand exceeds KV memory. If the running batch plus a desired admission would exceed available blocks, the scheduler must either refuse to admit or evict an existing sequence to free its blocks. Eviction can mean swapping a sequence's KV cache out to host memory to resume later, or recomputing it from the prompt when readmitted; either way the sequence's progress is preserved and it rejoins when memory frees. This policy is what keeps the system stable under overload instead of crashing on out-of-memory. The diagram shows the queue, the per-step scheduler, the running batch with paged KV, and the admit-on-finish loop.
End-to-end flow
Follow one request through a busy server. A chat prompt of 600 tokens arrives and joins the request queue behind others. The scheduler, on an upcoming iteration, sees that finished sequences have freed enough KV blocks to admit it, so it pulls the request from the queue and schedules its prefill.
The prefill pass runs the 600-token prompt through the model in one shot (or in a few chunks if chunked prefill is enabled to avoid a long stall), populating the request's KV cache across freshly-allocated paged blocks and producing the first output token. That first token is streamed to the user — this moment is the request's time-to-first-token, the sum of its queue wait and prefill time. The request is now a member of the running batch.
From here the request is in decode. On every step, the scheduler runs one forward pass across all active sequences; each, including ours, reads its KV cache, computes attention over its history plus the shared model weights, and emits exactly one new token, which is appended and streamed. Our request's KV cache grows by one block's worth of state every few tokens as it fills pages. Meanwhile, other sequences in the batch finish and depart, and new ones are prefilled and admitted around it — the batch's membership churns constantly while our request steadily streams its response.
Suppose midway through, a burst of new requests drives KV memory to exhaustion. The scheduler's preemption policy engages: it selects one or more sequences (perhaps by a policy like most-recently-admitted or lowest-priority) and evicts them, swapping their KV blocks to host memory or marking them for recompute, freeing GPU blocks to keep the highest-priority work progressing. If our request is preempted, its generated tokens are preserved; when memory frees, it is readmitted, its KV state restored or recomputed, and it resumes decoding from where it stopped — the user sees at most a brief pause, not a failure.
Eventually our request's model emits the end-of-sequence token. On that step the sequence is marked finished: its final token is streamed, it leaves the running batch, and its paged KV blocks are returned to the free pool. On the very next iteration the scheduler admits a waiting request into the freed capacity. There was no batch boundary to wait for, no idle padding slot held open for a slow sibling — the instant our request finished, its resources went to useful work. That per-iteration recycling, repeated across thousands of requests, is exactly why continuous batching keeps the GPU densely utilized and multiplies throughput over the static approach.