GPU Fine-Tuning Costs — Estimating Training Spend

The five numbers that decide the bill

Every fine-tuning estimate reduces to five quantities, and if you cannot name all five you do not have an estimate, you have a hope. They are: N, the parameter count of the model; D, the number of tokens you will actually process, which is your dataset size multiplied by epochs and inflated by padding; the method, which decides both the FLOP coefficient and the memory floor; the achieved throughput of one GPU on your specific shapes, in FLOP/s rather than in vendor peak; and the price of a GPU-hour on whatever SKU the memory floor forces you onto.

The skeleton is one line:

GPU-hours  =  (coef x N x D) / (achieved_FLOP_per_s x 3600)
cost       =  GPU-hours x GPUs_held / GPUs_computing x price_per_GPU_hour x runs

Almost every surprise on the invoice lives in the second line rather than the first. The FLOP term is usually small, tractable, and easy to compute in advance. What is neither small nor obvious is how many GPUs the memory floor obliges you to rent, how many of the hours you hold them are spent computing, and how many times you will run the job before you keep a result. This article prices a finite training job before you launch it. Steady-state serving spend and its attribution are separate problems, handled in GPU cost optimization and LLM FinOps.

Start with the FLOP bill — it is smaller than you expect

Start where the arithmetic is exact. A full training step costs roughly 6ND FLOPs: two per parameter per token in the forward pass, two more in the backward pass to produce input gradients, and two more to produce weight gradients. Freeze the base weights, as LoRA does, and the last term disappears, leaving about 4ND. Turn on full activation recomputation and you add another forward pass, taking full fine-tuning to roughly 8ND.

Now price a concrete job. A 7B model, a 40M-token instruction set, three epochs, so D = 120M tokens processed:

C = 6 x 7e9 x 1.2e8  =  5.04e18 FLOPs
at 400 TFLOP/s achieved:  5.04e18 / 4e14  =  12,600 s  =  3.5 GPU-hours

Three and a half GPU-hours. At an assumed $2.50 per GPU-hour that is under nine dollars of compute, and the number is not wrong. It is simply not the invoice, and the gap between the two is the subject of everything below.

One correction before you trust the FLOP term at long context. The 6ND form counts only parameter matmuls; attention itself scales with sequence length. With causal masking the extra work is about L x s x d / N as a fraction of the parameter term, for L layers, sequence length s and hidden size d. For a 7B model with 32 layers and d = 4096, that is roughly 8 percent at 4k context, 31 percent at 16k, and 61 percent at 32k. Below 8k you can ignore it. Above 16k you cannot, and an estimate built on short-context throughput numbers will be badly optimistic.

Memory sets the floor that FLOPs cannot

The FLOP bill says one GPU for a few hours. Memory says otherwise, and memory is what you actually rent. Count bytes per parameter of persistent state under standard mixed-precision training with Adam: a bf16 parameter (2), a bf16 gradient (2), and three fp32 optimizer tensors, the master weight, the first moment and the second moment (4 each). Sixteen bytes per parameter, of which twelve are optimizer state.

Eight-bit Adam quantizes the two moments to one byte each and takes the total to 10. LoRA keeps the base frozen at 2 bytes per parameter with no gradient and no optimizer state, and pays the full 16 only on the adapters, which are a fraction of a percent of N. QLoRA stores the base in NF4: 4 bits per weight plus about 0.127 bits per weight of quantization constants once double quantization is applied, so roughly 0.52 bytes per parameter. The mechanics of that format are covered in NF4 and QLoRA.

Activations sit on top. With full recomputation the persistent activation cost is about 2 x L x b x s x h bytes for the saved layer inputs, which for a 7B model at batch 1 and 4k context is close to 1 GB per sequence and scales linearly with microbatch size. Without recomputation it is roughly an order of magnitude more.

The minimum GPU count is then the ceiling of total bytes over usable HBM, where usable is perhaps 85 percent of nameplate after fragmentation and communication buffers. A 7B full fine-tune at 112 GB of state clears two 80 GB cards on capacity and in practice wants four once ZeRO sharding gathers and comms buffers are accounted for. That is the number that sets your bill: a job whose compute would fit in one GPU-hour budget now rents four cards for the whole wall clock.

LoRA and QLoRA are memory decisions with a cost consequence

LoRA and QLoRA are usually described as efficiency techniques, which invites the assumption that they cut the bill in proportion to the trainable parameter count. They do not, and the mismatch is the single most common error in fine-tuning budgets. What they change is the memory floor. What they barely change is the FLOP count.

Freezing the base skips only the weight-gradient matmuls, so a LoRA step costs about 4ND against a full step's 6ND. Backpropagation still runs through every layer, because the earliest adapter needs a gradient signal that only arrives by traversing the whole network. That is a 33 percent FLOP saving, or perhaps 1.5x wall-clock once you also drop the optimizer step and its memory traffic. It is not 10x, and any estimate built on a 10x assumption will be five to seven times low on compute.

QLoRA is stranger still: it is slower per token than bf16 LoRA on identical hardware, typically by 20 to 40 percent, because every matmul in forward and backward must dequantize NF4 blocks to bf16 first. QLoRA reduces cost only when the memory saving crosses a hardware boundary, taking you from two cards to one, or from an 80 GB SKU to a 24 GB one. If it does not cross a boundary, it is pure loss: same GPU, more hours, larger bill.

The same logic governs gradient checkpointing, which trades about 33 percent more FLOPs for roughly a tenfold cut in activation memory. Worth it if the freed memory lets you raise the microbatch enough to recover throughput or drop a GPU from the job. Otherwise it is a self-inflicted third added to the compute bill. The mathematics underneath these methods is developed in Fine-Tuning Math; the method-level overview lives in LLM fine-tuning architecture.

Utilization is the denominator, and where the dollars hide

Model FLOPs utilization is achieved model FLOP/s divided by the SKU's dense peak, and it sits in the denominator of the GPU-hours formula. Cost is therefore proportional to 1/MFU. Moving a job from 25 percent to 45 percent MFU cuts the compute bill by 44 percent, which is a larger single lever than any price negotiation you are likely to win.

Measure it rather than assume it. Take steady-state tokens per second per GPU from your own log and multiply by the coefficient: a 7B LoRA run at 12,000 tokens/s/GPU is 4 x 7e9 x 1.2e4 = 336 TFLOP/s, which against a roughly 990 TFLOP/s dense bf16 peak is about 34 percent. An estimate that assumed peak would have been three times optimistic before any other error.

Two effects dominate the gap for fine-tuning specifically. The first is padding. If your median example is 400 tokens and you pad every sequence to a 2048-token bucket, roughly 80 percent of the FLOPs you buy teach the model nothing, and you pay five times over for the same gradient signal. Sequence packing, which concatenates short examples into full-length sequences with correct attention masking, removes almost all of it. Track the useful token fraction, non-padding tokens over processed tokens, and treat it as a direct multiplier on the invoice.

The second is that fine-tuning datasets are small enough that fixed costs bite. A GPU blocked on CPU-side tokenization bills exactly the same as one running matmuls, and on a one-hour job a ten-minute compile-and-warmup phase is a sixth of the bill. Pre-tokenize to disk, keep the loader several batches ahead, and amortize warmup by not launching a fresh process for every configuration you want to try.

Three worked estimates, carried end to end

Three estimates carried end to end. Every rate below is an assumption chosen to make the arithmetic legible, not a quoted price, and the achieved throughput figures are illustrative planning numbers you should replace with measurements from your own run.

Read the first two rows against each other, because they contain the lesson. The QLoRA run on the cheap card costs more in absolute dollars and takes six times longer, despite the hourly rate being a third. QLoRA won the memory argument and lost the cost argument, because the 7B LoRA job already fit on one 80 GB card and never needed the saving.

The third row shows the other failure mode. Compute is 78 GPU-hours, but the job holds 32 GPUs for a four-hour wall clock covering provisioning, a 140 GB weight download per uncached node, NCCL initialization, two and a half hours of training and an evaluation pass. You are billed 128 GPU-hours for 78 hours of arithmetic, and the 40 percent you cannot see in the FLOP model is the part that scales with cluster size.

Interruptible capacity — the discount and the price of keeping it

Interruptible capacity is the largest discount available on training, and unlike serving it is genuinely usable, because a training job is restartable by construction. The question is what restartability costs you. Model it as an overhead factor:

F = 1 + lambda x (T/2 + R)      lambda = interruptions per hour for the JOB
                               T      = checkpoint interval, hours
                               R      = restart overhead, hours
take spot when  (1 - discount) x F  <  1

For a single-GPU LoRA run with an interruption rate of 0.05/hour, a 30-minute checkpoint interval and a 9-minute restart, F is about 1.02. Against a 65 percent discount the effective price is 0.36 of on-demand. That is not a close call.

Multi-node changes the sign of the argument, because for a synchronous job any single preemption stops everything: the job-level rate is the sum of the per-node rates. Thirty-two nodes at 0.02/hour each gives lambda = 0.64/hour, a kill roughly every 95 minutes. With a 30-minute interval and an 18-minute restart, F reaches 1.35, and R is optimistic because you must re-acquire 32 nodes that may no longer exist at that price.

Checkpoint size decides T, and it is where the method choice returns. A resumable full fine-tune checkpoint is the fp32 master weights plus both Adam moments, 12 bytes per parameter; gradients are never serialized. That is 84 GB for a 7B model and 840 GB for a 70B one, seven minutes of writing at an aggregate 2 GB/s, so you cannot checkpoint every ten minutes even if you want to. A resumable LoRA checkpoint at rank 16 on a 7B is about 17M trainable parameters, a few hundred megabytes with optimizer state, written in seconds. LoRA jobs are near-ideal spot workloads; large full fine-tunes are the workload spot was designed to punish.

Tokens and epochs — the only term that scales linearly

Tokens are the only term in the estimate that scales the bill linearly and that you control outright, so count them honestly. A dataset described as 50,000 examples is not an input to anything until you multiply by mean tokenized length, and the mean is what matters, not the median, because long examples dominate the sum. Then multiply by epochs, then divide by the useful token fraction from the packing discussion above. That product is D.

Epochs are pure multiplication and nothing about the third epoch is cheaper than the first. Supervised instruction tuning generally extracts most of its gain in one to three passes, after which you are paying full rate for memorization; a ten-epoch run is a decision to spend three times as much for a probably worse checkpoint. If you are unsure, run one epoch, evaluate, and continue from the checkpoint. Sequential runs cost the same as one long run and give you an exit.

Adapter rank, by contrast, is very nearly free and should not be the thing you economize on. On a 7B model with 32 layers at d = 4096, adapting the four attention projections gives 8.4M trainable parameters at rank 8 and 67M at rank 64. That is eight times the trainable parameters, but the adapter matmuls add only 2r/d to the cost of each adapted matrix, which works out to roughly one percent of total step FLOPs at rank 64 and well under half a percent at rank 8. Optimizer memory for the adapters goes from about 100 MB to about 800 MB, which no realistic job notices. Raise the rank, cut the epochs. How much data a fine-tune actually needs is a separate question, treated in Fine-Tuning Scaling.

Picking the SKU — dollars per unit of work, not dollars per hour

Instance selection is two divisions and one constraint, performed in that order. The constraint comes first: any SKU whose memory cannot hold state plus activations at a workable microbatch is not a candidate, no matter what it costs. The bytes-per-parameter table above answers that in one line.

Among survivors, compare cost per unit of work rather than cost per hour, because the two orderings differ often enough to matter. Divide price by achieved throughput:

card X:  $2.50/h  /  (400e12 x 3600 FLOPs/h)  =  $1.74 per 1e18 FLOPs
card Y:  $1.10/h  /  (130e12 x 3600 FLOPs/h)  =  $2.35 per 1e18 FLOPs

Card Y is 56 percent cheaper per hour and 35 percent more expensive per unit of work. It is still the right answer if the job fits on one Y and would need two X, because the memory constraint outranks the efficiency ratio. Both figures are illustrative, but the discipline of computing them is not optional.

Interconnect is the term people omit, and it only matters for one class of job. A single-node LoRA run is indifferent to NVLink. A sharded full fine-tune across many GPUs is not: on a PCIe-only chassis, parameter gathering and gradient reduction can halve achieved throughput, which doubles GPU-hours and erases a 30 percent hourly discount twice over. If the method is sharded, price the fabric as part of the SKU. If you can fit the job on one node, the entire question disappears, which is another quiet argument for the memory-efficient methods and for offloading optimizer state when it keeps you single-node.

Finally, availability is a cost. A cheaper region you queue three days for has spent more than it saved on any project with a deadline.

Why the estimate came in low

Compute estimates come in low with remarkable consistency, and always for the same short list of reasons. Work through it before you commit a number to anyone who will remember it.

You are billed for GPUs held, not GPUs computing. Node provisioning, container image pull, base weight download, compilation and warmup, and NCCL initialization all bill at the full cluster rate. On short jobs this is routinely 20 to 40 percent of the total.

Evaluation is not free. Generative evaluation is decode-bound and poorly parallelized, and a thorough eval suite run between epochs can rival an epoch of training in wall clock while achieving a small fraction of training MFU.

You will not run the job once. The first fine-tune of a new model and dataset pair fails: an out-of-memory at step 200, a learning rate that diverges, a malformed shard, a chat template that silently trained on the prompt. Budget two to three runs before the one you keep, and multiply again by every point of any sweep. A six-point learning-rate sweep is six times the single-run figure and no amount of MFU tuning changes that.

Reserved capacity idles between runs. Hours where a human is reading evaluation results bill identically to hours of training. On a reserved cluster this is often the largest single line item and it never appears in a FLOP model.

Storage accumulates quietly. Twenty retained checkpoints of a 7B full fine-tune is 1.7 TB of object storage; it is small against the GPU bill and large against zero, and nobody deletes it.

A defensible estimate therefore reads: FLOP-derived GPU-hours, divided by measured MFU, multiplied by the GPUs the memory floor forces you to hold, plus held-but-idle hours, multiplied by expected run count. The compute term is frequently the smallest of the three, and the practice of writing all three down is what separates an estimate from a guess. Whether to fine-tune at all rather than train from scratch, or to prompt a larger model instead, is a different comparison with different terms.