Why it matters

Chunked prefill smooths latency spikes from long prompts. Standard in modern serving.

Advertisement

The architecture

Split prompt into chunks.

Interleave with decode.

Balance TTFT and ITL.

Chunked prefillSplit promptinto chunksInterleavewith decodeBalanceTTFT vs ITLSarathi-Serve (Agrawal et al. 2023); vLLM implements; standard tech
Chunked prefill.
Advertisement

How it works end to end

Sarathi-Serve.

vLLM chunked-prefill.

Balance TTFT + ITL.