Batch Inference Deep Dive

Why it matters

Batch inference differs from online. Understanding shapes offline workloads.

The architecture

Very large batch sizes.

Sort by input length.

Prefill-heavy configuration.

Checkpoint recovery.

How it works end to end

vLLM offline API.

TRT-LLM offline mode.

Ray Data for scale.

Length bucketing.