Why it matters

Continuous batching drives throughput. Standard in vLLM, TGI, TRT-LLM.

Advertisement

The architecture

Per-iteration batch reformation.

Add ready requests.

Drop finished ones.

Continuous batchingPer-iteration batchreform each stepAdd readyrequestsDrop finishedsequencesOrca (Yu et al. 2022) introduced; 2-5x throughput vs static batching
Continuous batching.
Advertisement

How it works end to end

Orca 2022 introduced.

2-5x throughput.

Per-iteration batch reform.