Chunked Prefill: Stopping Long Prompts From Freezing Every Other Stream

The stall, in numbers

Without chunking, a server that runs continuous batching has two choices when a prompt arrives: run a prefill-only step and pause decoding (the classic vLLM behaviour before chunked prefill), or put the whole prompt into the next mixed step. Either way, the step that contains the prompt takes as long as the prompt's prefill. Using the lab's illustrative constants (0.035 ms of compute per token, attention that grows with the square of the prompt):

prompt tokens   prefill time (one step)   streams see a gap of
   2,000           ~ 70 ms + attention        ~ 75 ms
  24,000           ~ 840 ms + ~58 ms          ~ 0.9 s
  96,000           ~ 3.4 s + ~0.9 s           ~ 4.3 s

gap = 1.5 ms overhead + 0.035 ms * tokens + 2e-7 ms * tokens^2 / 2   (calculated, not measured)

Against a 30 ms TPOT target, a 0.9 s pause is about thirty tokens' worth of latency budget, charged to every stream on the GPU at once. When several long prompts queue up during a burst, an unchunked server may prefill them back to back or together, and the pause stretches to several seconds; the lab's first mission starts there, with a longest stall of 5.6 seconds and a p99 inter-token gap of 255 ms.

Why decode steps have room to spare

The reason chunking works so well is the roofline. A forward pass costs roughly the larger of two times: the time to move bytes (all weights, plus the KV cache being attended to) and the time to do the arithmetic (about two floating-point operations per parameter per token). A decode step for a modest batch processes few tokens, so it is bound by memory: the GPU streams 16 GB of weights for an 8B model in FP16 whether the batch holds one sequence or fifty.

# illustrative 8B-class model, FP16, ~2 TB/s memory, ~400 TFLOP/s effective
memory time per step   ~ 16 GB / 2 TB/s            = 8 ms   (independent of token count)
compute time per token ~ 2 * 8e9 FLOP / 400e12     = 0.04 ms
knee                   ~ 8 ms / 0.04 ms per token  = 200 tokens per step

below the knee: adding tokens to a step is nearly free (memory-bound)
above the knee: every token lengthens the step by ~0.04 ms (compute-bound)

A step with 32 decodes uses 32 of those 200 free token slots. Chunked prefill fills the rest with prompt tokens: the prefill work rides on a pass the GPU had to make anyway. The Sarathi paper called this piggybacking decodes on prefill chunks. The exact knee depends on the GPU, the model's size, its precision and the attention kernel, so treat 200 as an example: on recent data-centre GPUs and large models it is typically a few hundred tokens, and you find yours by measuring step time against tokens per step. The lab draws the knee as a dashed line on the token chart.

How the scheduler builds a step

With chunking on, the scheduler gives each step a token budget (vLLM calls it max_num_batched_tokens) and fills it in a fixed order. Decodes go first, one token for each running sequence, because they are latency-critical and cheap. Partially prefilled requests continue next, then new requests are admitted and their first chunk scheduled, until the budget is used:

def build_step(running, prefilling, waiting, budget, max_seqs):
    step = [(seq, 1) for seq in running]                 # every decode gets its token first
    left = budget - len(running)
    while waiting and len(running) + len(prefilling) < max_seqs:
        prefilling.append(waiting.pop(0))                # admit, KV memory permitting
    for seq in prefilling:
        if left <= 0:
            break
        n = min(left, seq.prompt_len - seq.prefilled)   # next slice of this prompt
        step.append((seq, n)); left -= n
    return step                                          # one forward pass, bounded size

Sarathi-Serve calls the result stall-free scheduling: no step ever contains more than a budget's worth of tokens, so no decode waits for more than one bounded step. The cost moves to the long prompt itself, which now finishes over many steps. A 24,000-token prompt with a 512-token budget and 30 running decodes needs about 50 steps of roughly 20 ms each, so its time to first token is around a second instead of 0.9 s, while nobody else's stream pauses for more than about 22 ms. DeepSpeed-FastGen's Dynamic SplitFuse uses the same idea, splitting long prompts and fusing short ones to keep each forward pass at a consistent size.

Picking the budget

The budget is a single number that sets two outcomes against each other: the duration of every step (and so inter-token latency for everyone) and the rate at which prompts are prefilled (and so time to first token for new requests). The lab's sweep shows the shape clearly on long-prompt RAG traffic at 4 requests per second:

These are outputs of the lab's synthetic model, not measurements. Below the knee, the budget is so small that prefill runs at a fraction of the GPU's capability and per-step overhead dominates; long prompts queue until their TTFT is unbounded. Far above the knee, chunking still exists but each step is long enough to break the TPOT target. The useful range starts a little above the knee and ends where the step time hits your TPOT budget: budget ≈ (TPOT_target − overhead) / compute_per_token, which with these constants is about 800 tokens for a 30 ms target. Then back off for headroom, because attention to long contexts and large decode batches add to every step.

Two refinements matter in practice. First, prefill throughput per step is highest just above the knee, so a server that is mostly prefill-bound (long prompts, short answers) wants a larger budget than one that is decode-bound. Second, giving one long prompt the entire prefill share of every step starves short prompts behind it. Recent vLLM versions can cap how many long prompts are partially prefilled at once and treat prompts above a threshold as long (look for options such as --max-num-partial-prefills and --long-prefill-token-threshold, and check your version's documentation), so a 96k-token prompt does not hold up a stream of short questions.

What a chunk costs that a full prefill does not

Chunking is not entirely free. Each chunk must attend to all prompt tokens before it, so the attention for chunk k reads the KV cache of chunks 1 to k−1 from memory again. The arithmetic for attention over the whole prompt is the same as unchunked prefill (it is still a triangle of query-key pairs), but the KV reads repeat once per chunk, and each step carries the scheduling and kernel-launch overhead. With very small chunks on very long prompts, these extra reads and overheads add up; the Sarathi-Serve authors report modest overhead at practical chunk sizes, which is why budgets of a few hundred to a few thousand tokens are the norm. The lab charges a fixed 1.5 ms per step and the attention term, which is why the 64-token budget collapses.

Chunked prefill also needs memory accounting that understands partial prompts: a request holds KV blocks for its prefilled slices across steps, so admitting many long prompts at once can consume the KV pool before any of them produces a token. Capping concurrent partial prefills helps here too.

The knobs in real servers

Names and defaults move between releases; check your version's documentation before relying on these.

In vLLM, when chunking is disabled the batched-token limit has to be at least the maximum model length, because a whole prompt must fit in one step; enabling chunking removes that constraint and the value becomes a latency knob. The default value has changed across versions and differs by usage context, so read what your deployment is actually running with from the startup log or configuration dump rather than assuming.

Interactions with prefix caching and disaggregation

Prefix caching shortens the part that needs chunking: if 20,000 of a 24,000-token prompt are already cached, only the last 4,000 tokens are prefilled, in about eight 512-token chunks. Chunk boundaries and cache block boundaries interact in some implementations, so a cache hit may be rounded down to a block boundary; the effect is small.

Disaggregated serving, covered in its own guide, removes prefill from decode GPUs entirely, so chunking stops being an inter-token latency tool there. It still matters inside the prefill pool, where a budget bounds how long one giant prompt can delay the next prompt's TTFT, and it lets the prefill instance stream KV to the decode side chunk by chunk. Many teams find that chunked prefill on colocated GPUs recovers most of the inter-token latency they wanted from disaggregation, without a second pool or a KV network, and only split when long prompts and tight SLOs push utilisation too low.

Measuring it in production

Chunking problems hide in averages, so instrument the tail. Export per-request inter-token gaps (or at least the longest gap per request) rather than only mean TPOT, and export the number of prompt tokens scheduled per step alongside step duration. Plotting step duration against tokens per step from live traffic gives you the roofline of your actual deployment, knee included, for free. Three signals are worth an alert. A p99 inter-token gap far above the step time you expect from your budget means something is bypassing chunking, often a server version where it is off or a configuration that disables it. A growing count of partially prefilled requests means prefill demand exceeds what the budget allows, and TTFT will follow. A rising number of steps that are almost entirely decodes, with prefill waiting, means the decode batch is crowding prefill out of the budget, which is the cue to cap running sequences or raise the budget. When you change the budget, compare goodput and p99 gap on a replay of the same traffic, because either metric alone can be improved by making the other worse.

A worked example: Belgavi's Book Store document questions

The hypothetical Belgavi's Book Store serves chat and document questions from the same GPUs. Chat prompts are short; one request in eight attaches a long document and carries 16,000 to 32,000 tokens. The product targets are TTFT under 2 seconds and smooth streaming under 30 ms per token. With chunking off, each document request freezes every chat stream on its GPU for most of a second, and in the lab's model only about 40% of requests meet both targets. With a 512-token budget, the longest pause any stream sees is about 22 ms and goodput rises to about 94%.

Then the team asks the capacity question: how much traffic can one GPU take? Mission four makes that concrete with long-document traffic and a 2× burst. Fix the scheduling first, then the budget, then raise the rate until one target breaks: in the lab's model the answer is about 2 requests per second per GPU, because the long documents' prefill work, not decode, is what saturates the GPU. That number tells the team the bottleneck is prefill compute, which is the signal to look at prefix caching for repeated documents or a separate prefill pool, rather than at decode settings.

Failure modes and common mistakes

• Leaving an unchunked default in place after the traffic mix shifts toward long prompts. Detect it by correlating p99 inter-token gap with the prompt lengths admitted in the same window.
• Setting the budget very small to minimise inter-token latency, then discovering that long prompts never reach their first token under load. Mission two starts from a 64-token budget.
• Setting it very large and believing chunking protects streams. A chunk of 8,192 tokens is a 290 ms step in the lab's model; chunking only bounds stalls to the budget you choose. Mission three shows this with a 96k-token prompt and a 16k budget.
• Tuning on average TPOT. Stalls live in the tail; watch p99 inter-token gap and the longest gap per request.
• Forgetting the budget also caps decode concurrency. Decodes consume one budget token each, so a 256-token budget cannot run more than 256 sequences, and leaves no room for prefill when it is nearly full of decodes.

Run the lab: Chunked prefill: slice the prompt, keep every stream moving

The lab simulates sixty seconds of synthetic traffic on one GPU, step by step. Each bar in the upper chart is one forward pass: its width is the step's duration, blue is decode tokens and amber is prefill tokens, with the compute knee drawn as a dashed line. The strip below shows each step's duration against the 30 ms TPOT target. Start with chunking off and long-prompt traffic on, then work through the missions.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Find out whether chunked prefill is enabled in the exact server version you run, and record the per-step token budget it actually uses.
2. Measure step time against tokens per step on your GPU and model to locate the knee where steps become compute-bound.
3. Set the budget from your TPOT target: the largest budget whose full step stays under the target, minus headroom for long contexts.
4. Track p99 inter-token gap and longest gap per request, and correlate spikes with the prompt lengths admitted at the same time.
5. If very long prompts mix with short ones, cap concurrent long partial prefills where your server supports it.
6. Load-test with your real prompt-length distribution plus a burst, and record the highest request rate that still meets both SLOs.

Sources and further reading

• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024)
• DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
• vLLM documentation: engine arguments
• Hugging Face Text Generation Inference: launcher arguments

Related in this series:

• Continuous Batching Explained: Why Static Batches Leave GPUs Idle
• Disaggregated Prefill and Decode: When Splitting the GPU Pool Pays Off
• TTFT, TPOT and Goodput: Setting LLM Latency SLOs That Mean Something

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.