Chunked Prefill in Serving — Token Budgets, SLO Sizing, and the Throughput Tax

The stall a long prompt creates

Iteration-level scheduling — the loop that admits and retires requests between forward passes rather than between batches — is assumed here; gpu_continuous_batching covers it. What that loop cannot fix on its own is the granularity of the work it schedules. A decode iteration advances every running sequence by exactly one token, so it is short and predictable. A prefill iteration processes an entire prompt, so its duration scales with prompt length.

Put those in the same queue and you get classic head-of-line blocking. The scheduler picks up a long prompt, dispatches it, and the GPU is busy for the duration of that one forward pass. Every sequence mid-generation produces zero tokens for that whole window. Users see it as a stutter: output flowing smoothly, then a long pause, then flowing again. The pause has nothing to do with their request. It is the tail of somebody else’s prompt, and it lands on whoever happened to be decoding when that prompt arrived.

The mechanism — one token budget per iteration

Chunked prefill replaces ‘schedule a prefill’ with ‘schedule a fixed number of tokens.’ Each iteration is given a token budget: the maximum number of token positions the batch may contain, regardless of which requests they belong to. The scheduler fills that budget in priority order, and decodes go first.

Every running sequence contributes exactly one token to the batch, so with B sequences decoding, B slots are spoken for. Whatever remains — budget T minus B — is handed to prefill: the next slice of the longest-waiting prompt, up to that many tokens. A prompt of P tokens therefore takes roughly P / (T - B) iterations to finish, spread across iterations that are also producing output for everyone else.

Note what the knob actually is. It is not ‘chunk size’ as a standalone constant — the chunk is the leftover. When the server is busy decoding, prefill automatically yields; when decodes drain, prefill automatically expands to fill the budget.

Sizing the budget from a TPOT SLO

The budget has a principled starting point, and it comes from the roofline. A decode-only iteration is memory-bandwidth-bound: it reads every model weight from HBM to compute a handful of token positions, so arithmetic intensity is dismal and the iteration time is essentially the time to stream the weights. Adding prefill tokens to that same batch adds FLOPs but reads the same weights once.

So there is a range in which extra prefill tokens are nearly free: until the added arithmetic takes longer than the weight streaming already did, iteration time barely moves. That crossover — the ridge point of the roofline for your model and your GPU — is the natural budget.

Derive it empirically rather than analytically. Sweep the budget, record iteration latency at each value, and you get a curve that is flat and then bends upward. Your inter-token latency target is a horizontal line across it. Pick the largest budget that stays under the line with headroom for the batch sizes you actually run, because iteration time is what TPOT is.

Whose latency moves — and in which direction

The sentence ‘chunking trades TTFT against TPOT’ is true, but it hides which request each number belongs to, and that is the entire point.

For the long prompt itself, chunking makes time-to-first-token worse. Its prefill now competes with decodes instead of pre-empting them, and it pays a per-iteration overhead several times over. Bigger chunks claw that back — fewer iterations, sooner to first token.

For every other request on the GPU, chunking makes things strictly better. Short requests that would have queued behind an atomic prefill now start almost immediately, so their TTFT improves dramatically; sequences mid-generation keep emitting tokens instead of freezing, so their TPOT stops spiking. Bigger chunks erode that benefit, because a longer iteration is a longer stall for everyone in it.

That is the real trade: one request’s first token against the whole batch’s smoothness. Serving a mixed workload, the batch almost always outweighs the individual.

The throughput tax — re-read KV and thinner GEMMs

Chunking is not free. The cost has two mechanical sources.

The first is attention re-reads. Queries in chunk k must attend to every key and value produced by chunks 1 through k. Those earlier entries are already in the KV cache, but they have to be pulled from HBM again on each subsequent chunk. One atomic prefill reads that state once; an n-chunk prefill reads growing prefixes of it n times, so KV traffic rises with the number of chunks. More chunks, more re-reading.

The second is shape. The projection and MLP GEMMs in a prefill have a row count equal to the token count. Shrinking that dimension lowers arithmetic intensity and gives the tensor cores less to hide latency behind, so each token costs marginally more than it would in a fat matmul.

Both push the same way: small chunks cost throughput. But mixing prefill tokens into otherwise decode-only iterations pushes the other way, because those iterations were wasting the tensor cores anyway. On genuinely mixed traffic the two effects often roughly cancel.

Chunk boundaries do not change the math

Does splitting a prompt change the result? It does not. Attention over a chunk is a partial computation of the same causal attention, not an approximation. Chunk k’s queries attend to all keys at positions at or before their own, which spans every earlier chunk in full plus a causal triangle within the current one. Nothing is truncated and no boundary is hidden from anything downstream.

What it demands is kernel support. The batch now contains sequences at wildly different query lengths — one token for each decode, hundreds for the prefill slice — so the attention kernel must be a variable-length one that takes per-sequence offsets, and the mask must be built from the chunk’s absolute position in the prompt, not its position within the chunk. An off-by-one there produces output that is plausible rather than obviously broken, which is why this belongs to the framework and not to your configuration file. Positional encodings likewise index on absolute position.

Prefix caching shortens what has to be chunked

Chunking cost is proportional to the number of prompt tokens that actually need computing, and that is not the same as prompt length. When a request shares a prefix with something already served — a system prompt, a few-shot preamble, an earlier turn of the same conversation — the KV entries for that prefix can be reused rather than recomputed. Only the uncached suffix enters the scheduler.

The interaction is a pleasant one. A 16k-token prompt whose first 15k are a cached system preamble is, to the chunked-prefill scheduler, a 1k-token prompt: possibly a single chunk, certainly not a head-of-line hazard. Workloads with heavy prefix sharing therefore need chunking far less often than their raw prompt lengths suggest.

Two consequences for tuning. Measure prompt-length distributions after cache lookup, not before, or you will size the budget for work that never happens; and expect that distribution to shift as hit rate does. The caching mechanism itself belongs to gpu_prefix_caching.

Chunked prefill or disaggregation

Both approaches attack the same interference, from opposite ends. Chunked prefill keeps one pool of GPUs and makes the interference small enough to tolerate by slicing the offender. Prefill-decode disaggregation removes the interference entirely by running prefill and decode on separate hardware and shipping the KV cache between them.

Chunking wins where its costs are low and disaggregation’s are high: a single cluster, a modest deployment, traffic that mixes short and long prompts unpredictably, and no appetite for operating two autoscaling pools plus a KV transfer path. It is a scheduler setting, not an architecture, and because the pool is shared it degrades gracefully: a burst of prefill borrows capacity rather than saturating a fixed-size tier.

Disaggregation wins at scale, when both SLOs are strict and the two phases want genuinely different parallelism and hardware. That decision, with its cost model and its KV-transfer requirements, lives in gpu_pd_disagg; treat this as the handoff rather than a second opinion.

Tuning it in production

Instrument both latencies by request class; the mechanism is a redistribution between them. The failure signature of a budget set too high is TPOT p99 spiking exactly when long prompts arrive — correlate the spikes with prompt-length percentiles before touching anything else. The signature of a budget set too low is TTFT climbing across the board while throughput sags, because per-iteration overhead is being paid too many times per prompt.

Re-tune after anything that moves the roofline: a different model, a different GPU, a quantisation change, or a tensor-parallel degree change, since all of them shift where the latency curve bends. And keep the budget honest about admission — chunking smooths the work you accepted, it does not decide what to accept, which is gpu_admission_control’s job. Admit more concurrent prefills than the budget can drain and they simply queue, and the stall reappears one layer up.