Every inference server has to answer one question before it answers any request: which requests run together in the same forward pass? There are four answers in common use — static, dynamic, continuous, and chunked prefill — and picking among them is not a matter of taste. Each is correct for a specific shape of workload and fails predictably on the wrong one. This is not a mechanism tour; the internals live in their own pieces. What follows is the selection logic: the five workload signals that decide the answer, a path from signal to strategy, and which corners each strategy still owns. The short version, stated up front so the rest can qualify it: continuous batching is the default for online generative serving, and you should need a reason to choose anything else.
The four strategies in one paragraph each
All four chase the same win. A decode step is memory-bandwidth bound — the GPU streams the whole weight set out of HBM to emit one token — so sequences that share a step share that read. The strategies differ only in when a request may join the batch, when it may leave, and what happens to everyone else while it runs.
Static batching fixes the batch up front: N requests go in, the batch runs to completion, nothing joins and nothing leaves. It is a for-loop over a dataset you already have.
Dynamic batching puts a queue in front of the model. The server accumulates arrivals until it hits a target batch size or a timeout expires, then dispatches that batch as an indivisible unit. The timeout is the entire design: a bounded wait traded for a fuller batch.
Continuous batching reschedules at every model iteration. A sequence that finishes releases its slot immediately and a waiting request takes it, without the rest of the batch pausing. The batch is a living set, not a shipment.
Chunked prefill is not a fourth assembly policy — it is a modifier riding on top of continuous batching, splitting a long prompt into bounded pieces so one giant prefill cannot monopolise an iteration and stall everyone else's decoding.
Axis 1 — is there a latency SLO, and which one
The first question is not which strategy is fastest. It is whether any single request has a deadline. If nobody is waiting — an overnight embedding job, a bulk classification pass, a synthetic-data run — per-request latency is not a metric, throughput per dollar is, and every mechanism that exists to protect individual latency is pure overhead.
If there is an SLO, split it in two. Time to first token is dominated by queueing plus prefill; time per output token is dominated by what else shares your decode iterations. They are optimised by different things, and a workload can violate one while comfortably meeting the other. A chat product with long system prompts usually has a TTFT problem; an agent streaming a long answer while other users' prompts land has a TPOT problem. Naming which one is being missed is usually enough to pick the fix, because static and dynamic batching both damage TPOT structurally, while chunked prefill exists specifically to protect it.
Axis 2 — arrival pattern
Is the work known or does it arrive? A corpus in a file is known: you control the order and the concurrency, and can size a batch once and reuse it forever. Online traffic arrives, and arrives unevenly.
For online traffic the number that matters is how many requests are in flight when a scheduling decision is made. At healthy load a continuous scheduler always has a queue to pull from, so slots refill instantly and no waiting policy is needed. At low, bursty QPS the same scheduler often finds an empty queue and runs iterations at batch size one, wasting most of the bandwidth it just spent reading weights. That sparse-arrival regime is the one case where deliberately waiting — a dynamic batcher's timeout — can beat work-conserving scheduling, because a few milliseconds of induced delay buys a batch several times larger. Above some request rate the timeout never fires and dynamic batching degenerates into a size-triggered dispatcher.
Axis 3 — variance in output length
This is the axis that eliminates most candidates, and the one people underweight. In any scheme where the batch runs to completion as a unit, it occupies the GPU for as long as its longest member. Nineteen requests that finish in 40 tokens sit in their slots doing nothing while the twentieth grinds through 2,000, because no slot is released until the whole batch retires.
The cost scales with the ratio of the longest to the median generation, and open-ended traffic routinely spans two orders of magnitude — a one-word confirmation and a full document review land in the same batch. Worse, output length is not knowable in advance, so you cannot sort your way out of it online as you can offline.
The test is simple. If every request costs exactly one forward pass — an embedding model, a cross-encoder reranker, a classifier, a vision model — variance is zero by construction and batch-as-a-unit schemes are fine. If generation length is open-ended, they are not, and no tuning recovers it.
Axis 4 — memory predictability
Batch policy and memory policy are the same decision viewed twice. A fixed batch of fixed shapes has a peak footprint you compute once, on paper, and then forget: pick the largest batch that fits, run it forever, never see an out-of-memory error. That predictability is worth real money when you are renting one GPU to chew through a queue unattended.
A living batch gives it up. Membership changes every iteration and each resident sequence's KV cache grows by a token per step, so the footprint is a moving target depending on decisions the scheduler has not made yet. Serving that safely needs two more mechanisms — a paged allocator so KV memory is not reserved for a length nobody reached, and an admission policy that refuses or preempts work before the allocator runs dry. Both are prerequisites rather than extras, and if your platform cannot supply them, that is a legitimate reason to stay simpler rather than run a scheduler that OOMs at peak.
Axis 5 — implementation complexity you actually pay
Ranked by what it takes to build: static is a loop; dynamic is a queue and a timer; continuous is a scheduler, a preemption story, and a paged KV allocator that must stay correct under memory pressure. Continuous looks expensive.
In practice you never pay that bill, because you are not implementing any of it — you are choosing a serving framework that already did. Modern inference stacks ship continuous batching as the default path, so the complexity is somebody else's and what you configure is a handful of knobs: maximum concurrent sequences, a memory fraction, a per-iteration token budget.
The real cost is now operational. A continuous scheduler makes latency load-dependent in ways a fixed batch does not, so you need per-request telemetry, queue-depth dashboards, and a load test that finds the knee before your users do. Budget for the observability, not the scheduler.
The decision path
Read top to bottom and stop at the first row that describes your workload; the earlier rows are the narrower claims.
| Workload signal | Strategy | Why |
|---|---|---|
| Whole corpus known, nobody waiting | Static | No SLO to protect; sortable offline; fixed, safe memory peak |
| One forward pass per request (embeddings, rerank, classify, vision) | Dynamic, small timeout | Zero length variance, so batch-as-a-unit costs nothing; the timeout fills batches at low QPS |
| Open-ended generation, requests arriving, any SLO | Continuous | Length variance and arrival jitter both demand iteration-level slot reuse |
| Already continuous, long prompts spiking inter-token latency | Add chunked prefill | Bounds how much prefill can occupy one iteration |
| Already continuous, running out of KV memory at peak | Not a batching change | Paged KV plus admission control; swapping strategy hides the symptom |
One row deserves plain words. Dynamic batching is not a weaker continuous batching; it is the right tool for a different model class, and it remains the standard policy in general-purpose model servers because most models outside text generation really do finish in one pass.
Where the non-default choices genuinely win
Static wins offline by more than people expect. With the full workload in hand you can group similar-length inputs so a batch wastes little work on padding, run at whatever concurrency saturates the device, and skip per-iteration scheduling entirely. For a fixed corpus and a deadline measured in hours, that simplicity is a feature.
Dynamic wins wherever a request is one forward pass — a large share of production ML: retrieval embeddings, rerankers, moderation classifiers, detection and segmentation models. It also wins the sparse-arrival corner above, where a small induced wait manufactures a batch that would not otherwise exist.
Chunked prefill wins when prompts are long and answers are streamed — RAG over large retrieved contexts, agents replaying transcripts, document question answering. It is not free: smaller pieces mean slightly less efficient matrix multiplications, and the chunked request's own first token can slip. That is the price of steadier decoding everywhere else.
How to confirm the choice instead of assuming it
Every claim above is a hypothesis about your traffic, and traffic is measurable. Three numbers settle most arguments: the distribution of prompt lengths, the distribution of output lengths, and the arrival rate over a real day rather than an average. High output variance with steady arrivals points at continuous before you benchmark anything; near-zero variance with spiky arrivals points at dynamic.
Then load-test against the SLO, not against a throughput number. Sweep the offered request rate, plot TTFT and TPOT percentiles alongside tokens per second, and find where the percentiles bend upward. The useful figure is goodput — requests per second that still met the SLO — because raw throughput keeps climbing well after users start noticing. Re-run the sweep whenever prompt shape changes: a longer system prompt or a bigger retrieval context can move a workload across one of these axes with nobody touching the server.