Static batching is the simplest way to keep a GPU busy: pick a batch size B, gather B inputs, pad them to a common shape, run the model, and hand back every result at once. Modern serving stacks have spent years replacing it, and for good reason — under an online workload a static batch is held hostage by its slowest sequence, and the GPU idles while finished rows are recomputed for nothing. But the conclusion inverts the moment nobody is waiting. In an offline job there is no latency budget to blow, only wall clock and cost, and static batching’s fixed shapes buy things a dynamic scheduler cannot: stable kernel selection, capturable launch sequences, predictable memory, and no per-step scheduling overhead. This piece makes the honest case for the old technique in the place it still belongs.
The shape that defines it
The defining property of a static batch is not its size — it is that membership and shape are fixed from admission to completion. You collect B inputs, pad them into one rectangular tensor of shape [B, L], and run the model over that rectangle. No new work joins mid-flight, and no finished row leaves early. The batch is a unit: it enters together and it exits together.
Everything good and everything bad about the technique falls out of that one commitment. Because the shape never changes, every kernel launch, every allocation and every scheduling decision can be made once and reused. Because membership never changes, a row that finished at step 100 keeps occupying a slot until step 900. Which of those two dominates is entirely a property of your workload, not of the batching strategy.
Why it loses online — the slowest sequence sets the clock
Autoregressive generation is where static batching falls apart. Output lengths are not known when you form the batch and they vary enormously: in a batch of 32 chat requests, thirty-one might finish in a hundred tokens and one might run to two thousand. The batch occupies the GPU for two thousand decode steps regardless. From step 101 onward you are launching full-width attention and MLP kernels in which thirty-one of thirty-two rows compute results that are immediately discarded.
Two things break at once. Throughput collapses, because most of the arithmetic is garbage. And latency becomes a lottery: a request’s completion time depends on the unluckiest co-tenant it happened to be batched with, so your p99 is set by other people’s prompts. That is exactly the pathology iteration-level scheduling exists to fix, and why gpu_continuous_batching is the correct default for anything user-facing.
Offline inverts the conclusion
Now delete the user. An evaluation sweep, a bulk embedding job or a synthetic-data run has no service level objective at all. Nobody is watching a spinner; the only metrics that exist are total wall clock for the whole dataset and tokens per dollar. ‘This request waited 40 seconds behind a long one’ is not a defect when the job runs for six hours and the results are consumed as a file.
More importantly, a large share of offline GPU work is not ragged in the first place. Embedding, classification, reranking, reward scoring and log-probability evaluation are prefill-shaped: you push tokens in and take one vector or one score out. There is no decode tail to be held hostage by, so the strongest argument against static batching simply does not apply. What remains is padding, and padding is a problem you can arithmetically solve.
Fixed shapes are a precondition, not a detail
A GPU library does real work to choose how to run a matrix multiply: tile sizes, split-k, epilogue fusion and algorithm choice are selected per problem shape, and autotuning caches are keyed on shape. Feed a kernel a new [B, L] every iteration and you re-pay selection, re-warm caches, and sometimes land on a worse algorithm. Feed it the same rectangle ten thousand times and you pay once.
Fixed shapes are also the entry ticket for CUDA graph capture, which replays a recorded launch sequence instead of re-issuing it from the host; reshaping every iteration forecloses that option outright — see gpu_cuda_streams_graphs for the mechanism. And a static loop does no host-side scheduling between steps at all, where a continuous-batching server runs an admission and eviction decision before every single decode step.
Memory you can actually predict
With a fixed batch and a fixed maximum length, the KV cache footprint is arithmetic, not a forecast: batch times layers times two times key/value heads times head dimension times length times bytes per element. You allocate it once, up front, and it never moves. There is no allocator churn, no fragmentation accumulating over hours, and no possibility of an out-of-memory event three hours into an eight-hour job because an unusually long request arrived.
That predictability is worth more than it sounds, because it changes how aggressively you can size the run. Dynamic serving has to leave headroom for a worst case it cannot see coming; an offline job with a known dataset can push the batch right up against device memory. Paged allocation exists precisely to make dynamic membership memory-safe — that is gpu_paged_kv_cache’s territory, and offline you often do not need it.
The padding arithmetic
Padding waste has an exact definition: you pay for every slot in the rectangle and use only the real tokens. Take an illustrative batch of 32 sequences whose longest member is 1024 tokens and whose mean length is 576. You pay for 32 × 1024 = 32,768 token-slots and use 18,432 of them. That is 56% useful work and 44% burned on padding — before any model-level inefficiency.
| Batching order | Slots paid | Slots used | Useful |
|---|---|---|---|
| Arbitrary order, pad to batch max | 32,768 | 18,432 | ~56% |
| Length-sorted, pad to batch max | ~18,600 | 18,432 | ~99% |
| Coarse buckets (256/512/1024) | ~23,000 | 18,432 | ~80% |
Figures are illustrative and depend entirely on your length distribution, but the ranking is robust: how you order the dataset is worth more than most kernel-level tuning you could do to the same job.
Sort by length — the cheapest fix there is
The fix is embarrassingly simple. Tokenize the whole dataset, sort by token count, then take consecutive runs of B. Because neighbours in a sorted list have nearly identical lengths, intra-batch spread shrinks roughly by the ratio of batch size to dataset size: 10,000 sequences spanning 128 to 1024 tokens, cut into 313 batches of 32, leaves each batch spanning about three tokens. Padding waste goes to roughly nothing for one tokenization pass and a sort.
Two cautions. Keep the original index alongside every row and reorder the outputs at the end — silently returning results in sorted order is a classic and very quiet bug. And be honest about the limit: sorting solves the prefill dimension, where length is known before you batch. It cannot sort by output length, because output length does not exist yet.
Bucketing and the shape-count trade
A full sort produces a different shape for nearly every batch, which throws away the fixed-shape benefits you came for: each new [B, L] re-pays kernel selection and needs its own captured graph. Bucketing is the compromise. Quantise lengths to a small ladder — 256, 512, 1024, 2048 — and pad each sequence up to its bucket. Now the job only ever sees four shapes, each one warmed and capturable.
The knob is the bucket count, and it is a straight trade: more buckets means less padding but more warmup, more autotune entries and more captured graphs held in memory. A refinement worth the ten lines it costs is to hold the token budget constant rather than the batch size, so the 256-token bucket runs a proportionally larger B than the 2048 one and every batch fills the same memory envelope.
When to reach for it
The workloads where static batching is the right default share a profile: the entire input set is known in advance, no human is blocked on any individual result, and success is measured in throughput per dollar. Evaluation and benchmark sweeps, bulk embedding of a corpus, dataset labelling and classification, offline reranking, reward-model scoring, and synthetic-data generation with a bounded output cap all fit cleanly.
There is an operational argument too, and it is underrated. A static batch job is a sorted loop over a dataset: deterministic, trivially resumable from an index, reproducible run to run, and debuggable with a print statement. A continuous-batching server is a scheduler you now have to operate, monitor and reason about. For a one-off job that has to finish correctly tonight, the boring loop is often the better engineering decision.
Where the offline case breaks down
The case is honest only if you say where it fails. If your offline job is long-form generation with wildly variable output lengths — agent rollouts, long synthesis, anything with early stopping — the ragged decode tail is back and it will eat the entire fixed-shape advantage. There the answer is to run continuous batching offline as well; the technique is not exclusively an online one, it is just mandatory there.
Measure rather than assume. The number to watch is the fraction of rows still active over the life of a batch: if it decays quickly, you are paying for empty slots and should switch strategies. Watch achieved memory bandwidth too — if a padded batch is bandwidth-bound moving zeros, sorting will buy you more than any kernel change.