A shared GPU fleet has one invoice and many spenders. LLM FinOps is the discipline that turns that single number into an answer to a specific question: who caused this cost, and who should carry it? That is a measurement and allocation problem, not an efficiency problem — the levers that shrink the bill only get pulled once somebody can see their own line in the total. This piece covers the accounting side: the unit-cost model and why achieved utilization sits in its denominator, what to capture at request time, how to roll tokens up by team, feature and customer, what to do with the residual nobody requested, and the reporting rhythm that changes behaviour.
The billable unit is a GPU-hour, not a token
If you run your own fleet, nothing in your cost base is denominated in tokens. You pay for wall-clock time on accelerators plus the overhead attached to each one. Tokens are a product unit; GPU-hours are the cost unit, and all of LLM FinOps lives in the conversion between them.
Write the fully-loaded rate for one hour of one GPU as R. Not the sticker rate: fold in the share of network fabric, the storage holding weights and checkpoints, control-plane nodes, and amortised platform-team cost. Two of the three FinOps phases below — attribution and governance — are this article’s subject; the middle phase, the levers that shrink R or raise throughput, is deliberately left alone here.
Utilization sits in the denominator
Let T be the tokens one GPU delivers per hour when saturated, and u be achieved utilization — the fraction of purchased GPU-hours actually spent on requested work. Cost per 1,000 tokens is 1000 × R / (u × T). The floor 1000R/T is a hardware and software property; everything above it is an operations property, and it multiplies:
| Utilization u | Cost per 1k tokens |
|---|---|
| 1.00 (saturated) | 1000R/T |
| 0.50 | 2000R/T |
| 0.25 | 4000R/T |
| 0.10 | 10000R/T |
This is why a published per-token price is a terrible internal accounting constant. Your own unit cost is a moving quantity that a quiet weekend, a capacity pre-buy, or a badly packed batch can shift by an order of magnitude. Recompute it per period from measured GPU-hours and measured tokens; never quote last quarter’s figure as a rate card.
Per token is not one unit
A single blended price per token quietly cross-subsidises workloads, because the two halves of an inference request have different marginal cost on the same hardware. Prefill processes the whole prompt in parallel and is compute-bound, running the tensor cores hard for a short burst. Decode emits one token per step per sequence and is dominated by reading weights and KV cache out of HBM, so it is bandwidth-bound and sequential.
So an input token and an output token consume different amounts of the resource you actually pay for, and the ratio depends on your batch composition, not on any price list. Cached prefix tokens complicate it further: a request hitting a warm prefix skips most of its prefill, so you must decide whether that saving accrues to the lucky requester or to the team that populated the cache. Meter prompt, generated, and cache-hit tokens as three counters. Blending them is a reporting decision you can make later; unblending them afterwards is impossible.
Capture at request time or lose it forever
Attribution is only ever as good as the metadata attached to a request while it is in flight; reconstructing it afterwards from logs and guesswork is the most common failure in LLM cost programmes. Treat these as required fields on every inference call, propagated through the gateway into whatever the serving layer emits:
| Field | Answers |
|---|---|
| owning team / cost centre | who gets the invoice line |
| product feature or endpoint | which capability is expensive |
| end-customer or tenant id | margin per account |
| environment: prod, eval, canary | is it revenue work at all |
| model, revision, precision | which config owns the cost |
| prompt / generated / cached tokens | the three cost units above |
| queue time and service time | demand versus congestion |
Enforce these at the gateway, not by convention: reject untagged requests in staging and flag them loudly in production, because every untagged call lands in the residual bucket somebody else ends up funding.
Rolling up by team, feature, and customer
Those three axes answer different questions and you need all of them. The team roll-up is what finance cares about; it must reconcile to the invoice, so it has to be a strict partition with no request belonging to two owners. The feature roll-up is what product cares about, because it exposes the endpoint that quietly costs more than the rest of the app combined. The customer roll-up decides pricing and account health.
Mechanically the roll-up is a weighted allocation: per period, take measured GPU-hours by pool, derive that pool’s unit cost from the tokens it actually delivered, then distribute that cost across tagged requests in proportion to their metered work. Keep tenant identifiers at low cardinality in the aggregate store — per-tenant time series outgrow a metrics system long before they outgrow a warehouse, so aggregate the long tail.
The residual: cost with no requester
Here is the number that separates a real cost model from a spreadsheet: the sum of attributed request cost never equals the invoice. Because utilization is in the denominator, everything that burned GPU-hours without serving a tagged request lands in a residual — idle time between peaks, warm pools held for latency, weight loading, cancelled generations, evaluation runs, canaries, and reserved capacity nobody used.
On a fleet sized for peak, the residual is no rounding error; it can rival the attributed cost. Pick a policy and publish it, because the policy is itself the behavioural lever. Spreading it pro rata by usage punishes efficient teams for someone else’s spikiness. Absorbing it into a platform overhead line keeps team numbers clean but hides the cost of headroom. Billing it to whoever demanded the headroom is the most accurate and the most argued-about. Whichever you choose, show the residual as its own line rather than smearing it.
Chargeback vs showback — the behavioural difference
Showback tells a team what it spent. Chargeback moves money out of that team’s budget. The reports can be byte-identical; the behaviour is not, because only one creates a trade-off against something the team already wanted.
Showback is the correct starting point, and skipping it is a classic mistake. Run it until the numbers survive contact with the owning engineers — if a team can still credibly say ‘that allocation is wrong,’ you are not ready to charge for it, and a disputed first invoice discredits the whole programme. Chargeback earns its machinery when GPU spend is large enough to change roadmap decisions and teams genuinely control their own demand. It backfires when they do not: charging a team for traffic driven by a central product surface produces arguments and gaming. A common middle position is chargeback for production serving, showback for experimentation, and a central budget for evaluation.
Committed capacity and who carries the risk
Most fleets mix long-term committed capacity, cheaper per hour and paid whether or not you use it, with on-demand or spot capacity, dearer per hour but only when consumed. Treat the mix as a portfolio decision: commitments cover the floor of demand you are confident about, and the flexible tier absorbs the variance above it. The floor is sized by the confidence interval on your own forecast, not by a target discount.
The attribution question is the one usually missed: who carries the commitment risk? Allocate committed hours at their discounted rate to whichever team fills them and you hand a windfall to whoever schedules earliest while pushing the unused remainder into the residual. Cleaner is to charge every team one blended internal rate reflecting the fleet’s committed and flexible mix, and let the platform team own the variance between that rate and the invoice. Consumers get a forecastable number, and purchasing risk sits with whoever makes the purchasing decision.
Budget enforcement and anomaly alerting
Budgets that only produce an email are decoration. Enforcement means a limit the platform can apply: a per-team token or GPU-hour quota at the gateway, priority tiers so a team over budget is served after teams within it, and a hard stop for non-production traffic. Because inference is metered per request, these are enforceable at admission time in a way most cloud budgets are not.
Alerting must be faster than the billing cycle. A runaway agent loop, a retry storm, or a prompt that grew a large context can multiply spend within hours, and a monthly report finds it three weeks late. Alert on rate of spend, compare each tag against its own recent baseline rather than a fleet-wide threshold, and watch the ratios that predict the bill: tokens per request, retry rate, cache hit rate, achieved utilization. A sharp drop in utilization is a cost incident even while the invoice looks unchanged, because it has raised your unit cost without yet raising your spend.
The reporting cadence that changes behaviour
Reports change behaviour when they arrive fast enough to connect a number to a decision somebody remembers making. Three loops do it. A near-real-time view, refreshed in minutes, exists for incidents: spend rate by tag, so an on-call engineer can find the runaway. A weekly view goes to the owning team, showing unit cost per 1k tokens and cost per business action — per conversation, per document, per resolved ticket — because that is the number an engineer can move. A monthly view reconciles to the invoice, shows the residual explicitly, and is the only one finance needs.
Two rules keep them useful. Always report unit cost alongside the absolute total, or a growing product looks like a failing one. And give every report a named owner expected to explain movements, because an unowned dashboard is read once.