Why it matters

In-flight batching is TRT-LLM's continuous batching. Same idea; NVIDIA branding.

Advertisement

The architecture

Same as continuous batching.

Different requests at different phase.

Per-step reform.

In-flight batchingPer-step reformbatch compositionMixed phasesprefill + decodeThroughput2-5x staticNVIDIA TRT-LLM naming for continuous batching; same underlying concept
In-flight batching.
Advertisement

How it works end to end

TRT-LLM implements.

Same as continuous.

Per-step reform.