Why it matters

Prefill and decode have different bottlenecks. Optimize each phase separately.

Advertisement

The architecture

Prefill: parallel, compute-bound, uses SM.

Decode: serial, memory-bound, uses HBM bw.

Prefill vs decodePrefillcompute-bound parallelDecodememory-bound serialOptimizeeach phase differentlyChunked prefill (Sarathi) interleaves phases for higher throughput
Prefill vs decode.
Advertisement

How it works end to end

Prefill batched large.

Decode continuous batching.

Chunked prefill blends.