Most GPU cost work fails for the same reason: the team optimizes the lever that is interesting rather than the one that is large. Kernel tuning, a quantization experiment, a spot-instance migration — each is real, and each is worth a fraction of what a fleet running at half utilization is silently burning. GPU spend is a product of three numbers: how many GPU-hours you hold, how much of each hour does useful work, and what you pay per hour. Levers that move the first two are multiplicative; levers that move the third are a discount on whatever you were already doing. This piece walks the levers in the order that actually pays, and frames each one the same way: expected win, what you give up, how to verify.
Order the levers before you pull any of them
Write your bill as hours × price × (1 / useful work per hour) and the ordering falls out. A lever that halves the price of an hour saves half of that hour. A lever that doubles the useful work inside every hour halves the whole fleet. And a lever applied to capacity you were not using saves nothing at all, because you are buying waste at a discount.
| # | Lever | Moves |
|---|---|---|
| 1 | Utilization | hours you actually need |
| 2 | Right-size model to task | work per request |
| 3 | Batching | work per GPU-second |
| 4 | Cache hit rate | work skipped entirely |
| 5 | Quantization | bytes moved per token |
| 6 | Request routing | work per request, dynamically |
| 7 | Context trimming | tokens per request |
| 8 | Commitment mix | price per hour |
Work down that list. Skipping to row eight is the most common and most expensive mistake in GPU finance.
Lever 1 — utilization, which dominates everything below it
Idle GPUs are the largest line item in almost every fleet, and they are invisible because they look like healthy capacity. Three numbers diverge and you must track all three: hours allocated to a team or job, hours the device is busy, and hours doing work someone asked for. Dev notebooks left running, training jobs holding eight GPUs while one preprocesses data, and inference replicas provisioned for a peak that arrives twice a day all sit in the gap.
Expected win: the arithmetic is brutal — a fleet at half utilization is paying twice per unit of work, so closing that gap is worth more than every other lever combined. What you give up: dedicated capacity and isolation; shared GPUs mean noisy neighbours and a wider blast radius. How to verify: allocated GPU-hours versus busy GPU-hours versus delivered tokens, per team, weekly — a ratio, not a dashboard of gauges.
Lever 2 — right-size the model and the SKU to the task
The default in most organizations is one large model behind one endpoint, serving classification, extraction, summarization and open-ended generation alike. Most of that traffic does not need the frontier model, and a meaningful slice does not need a flagship accelerator either: a smaller model that fits comfortably on a mid-tier inference part changes both the work per request and the class of hardware you rent.
Expected win: step-change, because cost scales with the parameters you read per token — not incremental. What you give up: a single-model architecture; you now own per-task evaluations and the risk of a quality regression you cannot see in aggregate metrics. How to verify: a task-level eval set with a pass bar per route, run before and after the swap. If you cannot measure quality per task, you cannot safely pull this lever, and building that harness is the first step.
Lever 3 — batching, until latency says stop
Decode is memory-bandwidth bound: reading the weights for one token costs nearly the same as reading them for many. Batching amortizes that read across concurrent sequences, which is why continuous, iteration-level batching is the single highest-leverage serving configuration change. The mechanics belong to the scheduler — see the continuous-batching and serving-architecture articles for how requests join and leave a running batch.
Expected win: large but saturating; throughput climbs steeply with batch size and then flattens once you are bandwidth- or memory-limited. What you give up: latency, specifically inter-token latency and the tail — a queued request waits for a scheduling slot. How to verify: plot tokens per second per GPU against p95 latency as you raise the batch ceiling, and stop at the knee your SLO allows, not at peak throughput.
Lever 4 — cache hit rate, the work you never do
The cheapest token is the one you do not compute. Two caches matter. Prefix or KV reuse skips prefill for the shared head of a prompt — a long system prompt, a fixed tool schema, the earlier turns of a conversation. An exact-response cache skips the request entirely for repeated queries. Paged KV cache and prefix sharing are covered in depth elsewhere; the cost lens is simply the hit rate.
Expected win: proportional to how much of your prompt is shared, and for agent and RAG workloads with fat static preambles that fraction is usually the majority of prefill. What you give up: HBM spent on cache instead of batch, plus routing constraints — sticky routing to the replica holding the prefix conflicts with even load balancing. How to verify: cached-token fraction and prefill tokens per request, tracked as first-class serving metrics.
Lever 5 — quantization, and the quality you pay for it
Lower-precision weights, and separately a lower-precision KV cache, shrink bytes moved per token. Because decode is bandwidth-bound, fewer bytes means directly faster decode; the freed memory also buys a larger batch or lets a model fit on a smaller or less numerous set of GPUs. The numeric formats, calibration and kernel support are the quantization articles' territory — here it is a cost lever with a quality price tag.
Expected win: substantial and compounding, since it feeds lever 3 by freeing memory for batch. What you give up: accuracy, unevenly — degradation concentrates in long-context, reasoning and rare-token behaviour that averaged benchmarks hide. How to verify: evaluate the quantized build on your traffic distribution, including the long tail, and confirm the kernels you deploy actually execute at the target precision rather than upcasting.
Lever 6 — route cheap requests to cheap models
Right-sizing is a static decision per workload; routing is the dynamic version. A classifier, a confidence threshold, or a small-model-first cascade sends easy requests to a small model and escalates only what needs the large one. Speculative decoding is the same instinct applied inside a single response, with a draft model proposing tokens the target model verifies.
Expected win: tracks the share of traffic that is genuinely easy, which in production mixes is often most of it. What you give up: a simple system. You add a router that can be wrong, and escalation means a request pays twice — small-model latency plus large-model latency. How to verify: escalation rate, end-to-end quality on the routed mix rather than per-model, and cost per resolved request — not cost per call, which a router flatters by inflating call count.
Lever 7 — trim the prompt and the context
Prefill cost is roughly linear in prompt tokens for the feed-forward work and worse than linear for attention, so context length is a cost dial that product teams turn without noticing. The usual offenders are a system prompt that grew by accretion, few-shot examples nobody re-measured after a model upgrade, a RAG stage that retrieves twenty chunks because ten felt risky, and full conversation replay instead of summarization.
Expected win: real but bounded — you are shaving a coefficient, not changing the shape of the curve. What you give up: recall and robustness; the fifteenth retrieved chunk occasionally is the answer. How to verify: track the distribution of input tokens per request, not the mean, then A/B the trimmed prompt against task quality. Do this after caching, since trimming a cached prefix can lower the hit rate and cost you more than it saves.
Lever 8 — commitment mix, on-demand, spot
Only now does procurement pay. Committed or reserved capacity discounts a GPU-hour in exchange for a term; spot and preemptible capacity discounts it further in exchange for the right to take it back. The correct order matters because commitment coverage should be sized to your steady, verified baseline — commit to a fleet you have not yet made efficient and you lock in the waste for the length of the term.
Expected win: a straight discount on the hour, no more. What you give up: flexibility and engineering time — spot demands checkpointing, drain handlers and requeue logic, which suits batch training and offline inference far better than latency-sensitive serving. How to verify: commitment coverage against actual busy hours, and preemption rate against work lost per preemption.
Knowing when you are at the floor
Every lever above eventually hits a wall you cannot tune past. In decode, the floor is memory bandwidth: if the batch is large enough to amortize weight reads and you are already moving close to the achievable HBM bandwidth of the part, the kernel is not slow — it is finished. Roofline reasoning is what tells you this: measure achieved bytes per second and compare it to the device's practical ceiling.
At that point the only remaining moves are structural — move fewer bytes (quantization, sparsity, a sparser architecture), reuse more (cache), or ask for less (routing, trimming). More kernel micro-optimization, another scheduler flag, another framework swap will return nothing, and the engineering hours spent are themselves a cost. Recognizing the floor is a cost lever too: it is the one that stops you spending salary to chase percentages that are not there.