Core concept
Continuous batching + PagedAttention enable variable-batch serving.
Advertisement
How it works
Latency-throughput trade-off. Higher batch = better throughput + worse per-query latency.
Advertisement
Trade-offs + gotchas
Every serving deployment. Match batch size to SLA.