Most inference tuning goes wrong the same way: somebody reads that 4-bit quantization is fast, quantizes the model, measures nothing, and ships a stack that is still slow because the real problem was a batch size of one. Inference levers are neither interchangeable nor additive. Each removes a specific kind of work, and if that work was not your bottleneck the lever buys nothing while costing quality, memory, or complexity. This is a triage guide: the levers in the order a serving team should try them, what each is worth, what each costs, and how to prove the win. It ends at the memory-bandwidth floor, where the hardware rather than your configuration is the answer.
Going faster is not the same problem as spending less
One scoping sentence first, because these goals get conflated constantly: this article optimizes latency and throughput — going faster on the hardware you have — while cost optimization is about spending fewer dollars per token, which brings in spot capacity, instance selection, autoscaling, and reserved commitments that have nothing to do with how fast a kernel runs.
The two overlap because throughput is the numerator of cost-per-token, so many speed levers are also cost levers. But they diverge at the edges: doubling batch size raises throughput and cuts cost per token while making individual requests slower, and a second GPU bought purely to hold a bigger KV cache improves latency and raises cost. Decide which objective you are serving, then write the target down as a number — a p95 time-to-first-token budget, a per-stream tokens-per-second floor, a tokens-per-second-per-GPU target. Untargeted optimization is how teams ship regressions they cannot detect.
Lever 0 — measure: are you prefill-bound or decode-bound?
Nothing below this line matters until you know which half of the workload you are in, because prefill and decode are different machines on the same silicon. Prefill processes the whole prompt at once: a matrix-matrix problem that saturates the tensor cores, compute-bound. Decode produces one token per step per sequence: a matrix-vector problem that reads every weight to do a tiny amount of arithmetic, memory-bandwidth-bound.
The split follows your traffic shape. Long prompts with short answers (retrieval-augmented QA, classification, summarization) are prefill-dominated, and the pain shows up as time-to-first-token. Short prompts with long answers (chat, agents, code generation) are decode-dominated, and the pain is inter-token latency. Get the ratio from your own request logs, then confirm with a profiler timeline. Diagnosis is its own discipline — this guide assumes you have done it and orders the fixes.
Lever 1 — batching, because idle silicon is the biggest waste
When it applies: always, and first, unless you already run continuous batching. What it buys: the largest throughput improvement available to most deployments, often a multiple rather than a percentage, because decode at batch size one leaves the tensor cores nearly idle. Weights are read from HBM every step regardless of how many sequences you serve; batching amortizes that read across all of them, so the marginal sequence is nearly free until you run out of bandwidth or cache space.
What it costs: per-request latency variance, and a scheduler you now have to reason about. Static batching is the wrong version — short requests wait for the longest one in the batch. Iteration-level (continuous) batching admits and retires sequences every decode step and is the mechanism you want; it has its own treatment elsewhere on this site. How to verify: plot tokens per second against concurrency and find the knee where p95 latency leaves your budget. That knee, not peak throughput, is your operating point.
Lever 2 — KV-cache management, because it sets your batch ceiling
When it applies: immediately after batching, because the KV cache is what stops you batching further. Every active sequence holds keys and values for every token it has seen in every layer, memory that grows linearly with context length and batch size. Once the cache fills the GPU, concurrency stops and the throughput you just unlocked evaporates.
What it buys: not raw speed — concurrency. Paged allocation eliminates the internal fragmentation of contiguous per-sequence buffers and substantially raises how many sequences you can hold; prefix reuse lets requests sharing a system prompt share its cache entirely, close to free throughput for chat and agent workloads with fixed preambles. What it costs: engine complexity, and for KV quantization a measurable quality risk on long contexts. How to verify: track how many concurrent sequences you sustain before preemption begins — a non-zero preemption rate means the cache, not compute, is your limit.
Lever 3 — quantization, priced in bytes moved per token
When it applies: when you are decode-bound. This is the lever people reach for first and it belongs third, because its mechanism is narrow: decode speed is roughly bytes-of-weights read per step divided by achievable HBM bandwidth, so halving the bytes roughly halves that term. That is the whole theory, and it is why quantization does little for a prefill-bound, compute-saturated workload.
What it costs: quality, unevenly. Weight-only 8-bit schemes are usually close to lossless; 4-bit needs good grouping and calibration data and degrades unevenly across tasks; quantizing activations too is faster still but far more sensitive. You also inherit a kernel dependency — a format with no fast kernel for your architecture can be slower than the baseline. How to verify: a task-specific eval on your own traffic, not a public leaderboard, plus a check that the quantized path uses a fused low-precision kernel rather than dequantizing to 16-bit first.
Lever 4 — kernels and engine choice, where the free wins hide
When it applies: once the algorithmic levers are in place and you are still leaving hardware on the table. What it buys: typically a solid percentage rather than a multiple, from three sources. Fused attention kernels keep the score matrix in on-chip memory instead of writing it to HBM, removing traffic that scales with sequence length squared. Fusing elementwise chains into their neighbours removes round trips for operations that do almost no arithmetic. CUDA graphs replay a captured launch sequence in one call, which matters in decode, where each step is a chain of tiny kernels whose launch overhead can rival the kernels themselves.
What it costs: portability and build friction. A compiled engine is often pinned to a GPU architecture, a precision, and sometimes a shape range, and rebuilding becomes part of your deploy. How to verify: a profiler timeline — gaps between kernels mean launch overhead, and achieved bandwidth well under peak on memory-bound kernels means traffic you have not removed yet.
Lever 5 — speculative decoding, a low-batch specialist
When it applies: when you are decode-bound and running at low batch size with spare compute — single-user sessions, latency-critical endpoints, on-premise deployments with one stream. What it buys: reduced wall-clock latency per token by verifying several cheaply drafted tokens in one forward pass, spending idle tensor-core capacity to buy back bandwidth-limited steps. With a correct acceptance test the output is distributionally identical to standard decoding, which makes it unusually cheap in quality terms.
What it costs: compute you may not have, plus a draft model or draft head to train, host, and keep aligned with the target. The critical failure mode is that the benefit shrinks as batch size grows — a large batch already saturates compute, so verification work competes with real work and can make throughput worse. How to verify: measure acceptance rate and end-to-end latency at your actual production concurrency, never at batch size one alone.
Lever 6 — change the model, the honest last resort
When it applies: when the levers above are exhausted and you still miss the target. What it buys: the biggest wins on the list, because model size is the term every other lever only shaves. A smaller model, a distilled variant, or a mixture-of-experts design that activates a fraction of its parameters per token cuts the bytes-per-step number outright.
What it costs: capability, and the evaluation work to measure how much. Multi-GPU parallelism belongs here too: tensor parallelism splits each layer so every GPU reads a fraction of the weights, genuinely reducing per-GPU decode time, but it adds a collective on every layer and is only worth it over a fast interconnect. Pipeline parallelism raises aggregate throughput without helping single-stream latency. How to verify: route a slice of production traffic to the candidate and compare task metrics, not perplexity.
The order is the point
The sequence has a logic: measure so you know which resource is scarce; batch so the hardware is busy; manage the cache so batching can continue; shrink the bytes each step reads; remove the overhead around those reads; trade spare compute for latency; only then change what you run. Out of order, levers mislead — quantizing before batching produces a small win that hides a large one, and speculation added before the scheduler is fixed can measure as a regression under load.
They also stack sublinearly, which is why compound speedup claims rarely reproduce: each lever removes part of one bottleneck, and the next then operates on a smaller base. Change one variable at a time and re-measure, or you will not know which change to keep when the quality report lands.
The memory-bandwidth floor
There is a hard bottom to this list. In decode, every step must read the model's weights from HBM to produce one token per sequence, so the fastest possible step time is model bytes divided by achievable memory bandwidth. No scheduler, kernel, or engine beats that; it is a property of model size and hardware, not software. When measured step time approaches that ratio, you are done optimizing.
Only three things move the floor. Read fewer bytes: quantize further, or use a smaller or sparsely-activated model. Read them faster: newer parts with higher bandwidth. Or split the read: tensor parallelism, paid for with interconnect traffic. Everything else — batching, paging, prefix reuse, fusion, graphs, speculation — extracts more useful output per read, which is why it helps right up until the read is all that remains. Knowing where that line sits turns an open-ended tuning project into a finite one, and tells you when the next step is a purchase order rather than another config flag.