Why architecture matters here
The architecture matters because prefill and decode are not just different sizes of the same work — they stress opposite parts of the GPU. Prefill runs the attention and feed-forward math over the entire prompt in parallel, so it is compute-bound: it can keep the matrix units busy and its cost scales with prompt length. Decode processes a single new token per sequence but must read the whole accumulated KV cache to do it, so it is memory-bandwidth-bound: the arithmetic is trivial and the step is dominated by moving KV data. Run them in isolation and each wastes what the other needs — decode leaves compute idle, prefill leaves bandwidth idle. Interleaving them in one batch lets a memory-bound decode step absorb a compute-heavy prefill chunk almost for free, which is the core efficiency argument for chunking.
It matters because the two user-visible latencies it governs are the two everyone measures. Time-to-first-token (TTFT) is how long a user waits before anything appears — dominated by prefill. Inter-token latency (ITL) is how smoothly the answer streams after that — dominated by decode-step cadence. An unchunked server trades one against the other: prioritize prefill and existing streams stutter; prioritize decode and new requests hang. Chunked prefill is the mechanism that lets you bound both at once, because no single prefill can ever consume a whole step and stall the decoders, and decode always runs, so the tail of neither metric is hostage to the other.
It matters because it makes the scheduler's behavior a tunable rather than an emergent accident. The per-step token budget and the chunk size are explicit dials: a larger budget or chunk pushes more prefill work per step (better prompt throughput, but a heavier step that lengthens every decoder's ITL), a smaller one protects ITL at the cost of taking more steps to ingest a prompt (raising TTFT for long prompts). Because the trade-off is a knob and not a side effect of which request happened to arrive when, you can size it to your SLOs — smooth streaming for a chat product, fast first token for autocomplete — instead of accepting whatever the arrival pattern produces.
Finally, it matters because it changes how a fleet's capacity is planned. With monopolizing prefill, a burst of long-prompt requests causes ITL spikes that look like the server is overloaded even when average utilization is modest, so operators over-provision to hide the spikes. Chunked prefill smooths those spikes into steady, higher utilization: the same GPUs serve a mix of long-prompt and streaming traffic without the latency cliffs, so you can run hotter with predictable tails. Understanding the mechanism is understanding why a chunked scheduler both improves throughput and tightens latency at the same time, instead of trading one for the other the way most tuning does.
The architecture: every piece explained
Top row: a new prompt becomes chunks under a budget. A new request arrives with a long prompt of N tokens. Instead of prefilling it whole, the scheduler conceptually splits it into chunks of a fixed size — 512 tokens is a common choice — each of which can be prefilled independently by extending the request's KV cache. Every step the scheduler works against a token budget: a cap on how many total tokens (decode plus prefill) it will put through one batched forward pass on the GPU. The budget is what makes the step's cost, and therefore its latency, predictable regardless of how long the arriving prompts are.
Middle row: how a single step is assembled. First the scheduler schedules decode tokens — exactly one for each live sequence currently generating, because those must advance every step to keep streams flowing. Whatever budget remains it fills with a prefill chunk from a prompt still being ingested, taking as many of that prompt's not-yet-processed tokens as fit. The two combine into a mixed batch — a batch containing both one-token decode rows and a multi-token partial-prefill row — that the model runs in a single pass. As each chunk is processed the request's KV cache grows by that chunk's worth of key/value entries, so after enough steps the whole prompt is ingested and the request flips into normal decode.
Bottom-left: the TTFT guarantee. Because a prefill chunk only ever consumes the leftover budget after decode is served, and because it is bounded to a chunk, a giant prompt can no longer seize an entire step. TTFT stays bounded: the new request makes steady progress every step rather than waiting for a slot big enough to prefill it whole, and no single request can push another's first token arbitrarily far out. The prompt takes a few steps to finish prefilling, but each of those steps also did useful decode work.
Bottom-right and ops: the ITL guarantee and the dials. Because decode is scheduled first and runs every step, ITL stays smooth — existing streams never freeze behind a prefill, so their inter-token cadence stays steady even as long prompts arrive. The ops strip names the tuning surface: the chunk size versus the token budget (bigger chunks ingest prompts faster but make each step heavier), the prefill/decode ratio the budget implies (how aggressively you favor new prompts over stream smoothness), the KV memory headroom (every in-flight prompt's growing cache consumes GPU memory, which caps concurrency), and the compute-versus-memory-bound balance — the whole point is to pack compute-hungry prefill into steps that were memory-bound, so you tune to keep the GPU busy on both axes.
End-to-end flow
Trace one long-prompt request arriving into a server that already has a dozen sequences streaming, and follow it from arrival to its first token and into steady decode.
Arrival and admission: a request lands with a 2,000-token prompt. The scheduler does not try to prefill all 2,000 at once. It admits the request into the set of 'prompts being ingested,' reserves KV-cache space for it, and marks that it has 2,000 tokens still to process. Nothing about the twelve already-streaming sequences changes — they will keep decoding a token each per step throughout.
First mixed step: the scheduler builds a step. It schedules one decode token for each of the twelve live sequences — twelve tokens — then looks at the remaining token budget. With a budget of, say, 1,024 tokens per step, roughly 1,012 remain, so it carves a prefill chunk of the new prompt's first ~512 tokens (chunk size caps it below the raw remainder) and adds that partial-prefill row to the batch. One forward pass runs the mixed batch: the twelve decoders each emit their next token, and the new request's first 512 prompt tokens are processed, appending 512 entries to its KV cache. The compute-heavy prefill chunk rode inside a step whose decode work alone would have left the matrix units underused.
Second and third steps — finishing prefill: the next step repeats: twelve decode tokens plus the prompt's next ~512-token chunk. After roughly four such steps the whole 2,000-token prompt has been ingested, its KV cache is complete, and the model has produced the request's first output token. Crucially, across those four steps the twelve existing streams never paused — their ITL stayed flat — and the new request's TTFT was the cost of four ordinary mixed steps, not a wait for one enormous prefill-only step behind other traffic.
Into steady decode: with prefill done, the new request joins the decoding set. From now on it is scheduled first, alongside the others, one token per step, and the leftover budget each step goes to whatever next long prompt is being ingested. The KV caches of all finished-prefill sequences are read every step to decode; the server watches its KV memory headroom so it admits new prompts only while there is room, evicting or queuing when memory runs tight. Across the whole episode the GPU stayed busy on both compute (prefill chunks) and bandwidth (decode reads), TTFT for the new request stayed bounded, and ITL for the incumbents stayed smooth — the two goals a monopolizing prefill could not satisfy together.