Fine-tuning takes a pretrained model that already knows how to read and write and nudges it toward your task, your tone, your data. The naive way — full fine-tuning — updates every one of the model’s billions of weights, and the moment you write down the memory that requires, you understand why an entire sub-field exists to avoid it. The Adam optimizer alone wants roughly 16 bytes for every parameter in fp32: the weight, its gradient, and two optimizer moments. For a 7-billion-parameter model that is over 100 GB before you have stored a single activation. Parameter-efficient fine-tuning (PEFT), and above all LoRA, sidesteps the whole bill by freezing the pretrained weights and training a tiny low-rank update instead. This piece works the numbers: the exact memory ledger of full fine-tuning, the linear algebra behind LoRA’s W + BA decomposition and why a rank of 8 or 16 is enough, catastrophic forgetting, the learning rates and epochs that actually work, and a worked example you can carry to your own model — ending with QLoRA, adapters, and prefix-tuning, the siblings in the same family.
Two ways to adapt a pretrained model
Every fine-tune is a choice between two philosophies. Full fine-tuning unfreezes the entire network and lets gradient descent move all of it. It is the most expressive option — nothing is off-limits, so in principle it can reach any behavior the architecture supports — and for decades it was simply what ‘fine-tuning’ meant. Parameter-efficient fine-tuning (PEFT) freezes the pretrained weights and trains only a small set of new or selected parameters, typically well under 1% of the total.
The reason the second philosophy exists is not accuracy — on most tasks the two land within a whisker of each other — it is resource math. Full fine-tuning of a modern model needs the memory of a small cluster and produces a fresh multi-gigabyte checkpoint for every task. PEFT fits on a single consumer or workstation GPU and produces adapters measured in megabytes. To see why the gap is so violent you have to count bytes, not parameters, because the parameter count is only a quarter of the story. The rest is what an optimizer drags along behind each weight, and that is where we start.
The memory ledger of full fine-tuning
Training memory is not just ‘the model.’ For each trainable parameter, an Adam-style optimizer in fp32 must hold four things:
Per parameter, fp32 Adam:
weight w 4 bytes
gradient ∇w 4 bytes
1st moment (m) E[∇w] 4 bytes # Adam momentum
2nd moment (v) E[∇w^2] 4 bytes # Adam variance
--------------------------------------
TOTAL 16 bytes / paramThat 16 bytes per parameter is the number to memorize. The weight itself is only 4 of the 16 — the other 12 are training overhead that disappears the instant you stop training. Adam keeps a running mean (m) and a running mean-of-squares (v) of the gradient per weight so it can adapt the step size dimension by dimension; those two buffers are the same size as the model, twice over. Add the gradient buffer and you are carrying four copies of the parameter tensor. On top of the 16 bytes/param sit the activations saved for backprop, which scale with batch size and sequence length and can add tens of gigabytes of their own. Full fine-tuning’s reputation for being memory-hungry is entirely this ledger.
A worked memory example: fine-tuning 7B
Put numbers on it. Take a 7B-parameter model and full-fine-tune it with Adam in fp32:
N = 7e9 params
weights : 7e9 × 4 = 28 GB
gradients : 7e9 × 4 = 28 GB
Adam m : 7e9 × 4 = 28 GB
Adam v : 7e9 × 4 = 28 GB
-----------------------------------
states : 7e9 × 16 = 112 GB (+ activations)112 GB before a single activation — already past the 80 GB of a top-end data-center GPU, so you are forced into multi-GPU sharding (ZeRO, FSDP) just to begin. Mixed-precision training does not rescue you the way people hope: keeping bf16 weights and gradients (2 bytes each) but an fp32 master weight plus fp32 Adam moments still totals roughly 2 + 2 + 4 + 4 + 4 = 16 bytes/param, sometimes 18–20 once you count the fp32 master copy separately. The headline does not move. This 112 GB wall — for a model that is, by 2020s standards, small — is the single most important motivation for everything that follows.
Why the optimizer states dominate
It is worth dwelling on which bytes hurt, because it tells you where to cut. Of the 16 bytes/param, the raw weight is 4 and is unavoidable — you need the model. The gradient (4) exists only during the backward pass. The two Adam moments (8 combined) are pure optimizer bookkeeping. So three-quarters of the training memory is not the model at all; it is the cost of computing and adapting an update for every weight.
This reframes the problem. If most of the memory is spent maintaining an optimizer state per trainable weight, then the way to shrink it is not to compress the weights — it is to reduce the number of trainable weights. Freeze a parameter and it contributes 4 bytes (its value, which you had to store anyway) and zero optimizer overhead: no gradient, no m, no v. If you can train just 0.2% of the parameters and freeze the rest, the 12 bytes/param of optimizer overhead collapses by the same factor. That is the entire thesis of PEFT, and LoRA is its cleanest expression: keep the huge pretrained weight matrix frozen, and route all the learning through a handful of tiny new parameters bolted alongside it.
LoRA: the low-rank update W + BA
LoRA (Low-Rank Adaptation, Hu et al. 2021) starts from a simple observation: fine-tuning changes a weight matrix from W to W + ΔW, and you do not have to store ΔW as a full matrix. You can factor it. Freeze W and represent the update as a product of two skinny matrices:
W ∈ ℝ^(d × k) # frozen pretrained weight
ΔW = B · A # low-rank update
B ∈ ℝ^(d × r) # 'down' -> 'up', init 0
A ∈ ℝ^(r × k) # init random Gaussian
r « min(d, k) # rank, typically 8..64
h = x W + (α/r) · x (B A) # forward passDuring training only A and B receive gradients; W never moves. Because B is initialized to zero, BA = 0 at step 0, so the fine-tune starts exactly at the pretrained model and departs smoothly — no shock to the network. The scalar α/r is a fixed scaling that decouples the learning rate from your choice of rank. The forward pass gains only one extra skinny matmul; the expensive x W is unchanged.
Counting LoRA parameters: 2 · r · d
The whole payoff is in the parameter count. A full update ΔW for a d × k matrix has d · k free numbers. The factored version has:
params(B A) = params(B) + params(A)
= d·r + r·k
= r (d + k)
for a square matrix d = k:
= 2 · r · dCompare 2·r·d against the full d^2. The ratio is 2r/d — for a typical hidden size d = 4096 and rank r = 8, that is 16/4096 = 1/256. A single attention projection drops from 4096×4096 ≈ 16.8M trainable parameters to 2·8·4096 = 65,536 — about 0.39%, a 256× reduction, for that one matrix. Rank is a dial you turn directly against capacity: r sets exactly how many degrees of freedom the update is allowed. Small r means a tiny, cheap adapter that may underfit a hard task; larger r costs more but can express a richer change. Crucially the cost is linear in r, so the knob is smooth and predictable.
Why a low-rank update is enough
The obvious objection is that BA can only ever be a rank-r matrix — a drastically restricted subset of all possible ΔW. Why should that suffice? The empirical and theoretical answer is that the update a fine-tune needs is itself low-rank. Pretraining has already learned rich, general features; adapting to a downstream task is a comparatively small, structured rotation of those features, not a from-scratch relearning. Aghajanyan et al. showed pretrained models have a low intrinsic dimension — you can fine-tune them well within a subspace of only a few hundred dimensions — and Hu et al. found that the fine-tuning delta’s own singular-value spectrum is dominated by a handful of directions.
In linear-algebra terms: the best rank-r approximation of ΔW (its top-r singular components) already captures most of the useful change, and the tail contributes diminishing returns. That is why raising r from 8 to 32 to 64 helps less and less — you are adding singular directions that carry little signal. LoRA is not a lossy hack that happens to work; it is a bet on a real property of pretrained networks, and the bet pays off across a wide range of tasks.
Scaling, initialization, and the α/r knob
Two details make LoRA behave. First, initialization: A is drawn from a small random Gaussian and B is set to exactly zero. The asymmetry matters — if both were random the update would start as noise and jolt the model; if both were zero no gradient could ever flow into A. Zeroing only B gives BA = 0 at the start (so training begins at the pretrained model) while still letting gradients reach both factors on the first step.
Second, the scaling factor α/r. The adapter output is multiplied by α/r before it is added to xW. Dividing by r means that when you change the rank you do not have to re-tune the learning rate — the effective magnitude of the update stays roughly constant. In practice people fix α (often to the same value as r, or to 2r) and treat it as a second capacity/strength control. A useful mental model: r sets how many directions the update can use, and α/r sets how hard it is allowed to push along them.
A worked memory example: LoRA on 7B
Return to the 7B model, now with LoRA on the attention projections. Suppose you adapt the four projections (q, k, v, o) in each of 32 layers, with d = 4096 and r = 16:
per matrix : 2 · 16 · 4096 = 131,072 params
matrices : 4 proj × 32 layers = 128
trainable : 128 × 131,072 ≈ 16.8M params (~0.24% of 7B)
optimizer memory (16 B/param, fp32):
16.8e6 × 16 ≈ 0.27 GB vs. 112 GB for full FT
frozen base still resident:
fp32 : 28 GB | bf16 : 14 GB | 4-bit (QLoRA): ~3.5 GBThe training-state cost falls from 112 GB to under a gigabyte — a ~400× cut in the part that scales with trainable parameters. The one cost you cannot escape is holding the frozen base weights in memory, because the forward pass still runs through them. That is exactly the residual QLoRA attacks by storing those frozen weights in 4-bit, which is what finally puts 7B fine-tuning inside a single consumer GPU.
Catastrophic forgetting
Full fine-tuning has a failure mode beyond memory: catastrophic forgetting. When every weight is free to move and you train hard on a narrow dataset, the model overwrites general capabilities it learned during pretraining — it gets better at your task while quietly getting worse at reasoning, formatting, or following instructions it used to handle. The pretrained knowledge lives in the same weights you are now stomping on, so a strong-enough update erases it.
LoRA is structurally more resistant. The pretrained weights W are frozen and therefore literally cannot be forgotten; all adaptation is an additive, low-rank side channel. The base model is preserved byte-for-byte, and you can even detach the adapter to recover the original network exactly. This does not make forgetting impossible — a large, aggressive adapter on a tiny dataset can still skew behavior — but the blast radius is far smaller. It is also why LoRA adapters compose so well operationally: one frozen base can host many swappable task adapters, each a few megabytes, instead of many full multi-gigabyte checkpoints that each drifted away from the original in their own direction.
Learning rate and epochs: full FT vs LoRA
Fine-tuning lives or dies on two hyperparameters, and full FT and LoRA want different values. Full fine-tuning demands a small learning rate — commonly 1e-5 to 5e-5 — precisely to limit forgetting: every weight is load-bearing, so a large step shatters delicate pretrained structure. It also wants few epochs, often just 1–3 passes; a pretrained model is already close to a good solution, and over-training on a small dataset is a fast route to memorization and lost generality.
LoRA tolerates a markedly higher learning rate — often 1e-4 to 3e-4, an order of magnitude above full FT. It can afford to because the frozen base is a safety net: the update is confined to a low-rank subspace, so a bigger step cannot wander as far off the manifold. The same 1–3-epoch guidance applies, watched with a validation curve rather than a fixed schedule — stop when held-out loss flattens. Two rules travel across both regimes: warm up the learning rate over the first few percent of steps, and prefer more, more-diverse data over more epochs on the same small set. Repetition teaches the model to recite; variety teaches it to generalize.
Where LoRA is applied inside the transformer
You do not have to LoRA-adapt every matrix, and choosing where to inject the adapters is part of the craft. The original work found the biggest returns from adapting the attention projections — the query, key, value, and output weights (W_Q, W_K, W_V, W_O) — and often just W_Q and W_V is enough to recover most of the task gain at half the adapter size.
Later practice frequently extends LoRA to the large feed-forward (MLP) matrices as well, which hold the bulk of a transformer’s parameters and can matter for tasks that need new knowledge rather than new behavior. The trade is the familiar one: more injection points and higher rank mean a more expressive adapter and a larger trainable footprint. A sensible default is attention-only at r = 8–16, then widen to the MLP or raise the rank only if the validation curve says the adapter is capacity-starved. Because every choice here moves the r(d + k) parameter count in a way you can compute in advance, you can budget the adapter precisely before you launch a single training step.
Merging the adapter back for inference
A subtle virtue of the additive form h = xW + (α/r)x(BA) is what happens at deployment. During training the adapter is a separate branch, which is why it is cheap. But once training is done you can fold it in: compute W' = W + (α/r)BA once, offline, and ship W' as an ordinary weight matrix.
The consequence is that a merged LoRA model has zero inference overhead — no extra matmul, no extra latency, identical shape to the base model. This is a genuine advantage over some other PEFT methods (adapters, prefix-tuning) that add modules or tokens the forward pass must always execute. You get the training cheapness of a side branch and, if you want it, the inference cleanliness of a plain fine-tune. The flip side is a choice: merge and you gain speed but lose the ability to hot-swap; keep the adapter separate and you can serve many tasks from one resident base by attaching a different few-megabyte adapter per request. Latency-critical single-task deployments merge; multi-tenant serving keeps them detached.
QLoRA, adapters, and prefix-tuning: the siblings
LoRA is the most popular member of a family. QLoRA is its direct extension for memory-bound setups: quantize the frozen base weights to 4-bit (the NF4 format), keep the LoRA factors in bf16, and back-propagate through the dequantized weights. With double-quantization and paged optimizers it fits a 7B fine-tune into roughly 6 GB and even 65B-class models onto a single 48 GB card — the base is tiny in 4-bit and the trainable adapter is tinier still.
The older adapter approach inserts small trainable bottleneck layers (down-project, nonlinearity, up-project) between the frozen sublayers; it works but adds a permanent forward-pass cost that cannot be merged away. Prefix-tuning and prompt-tuning take a different route entirely: freeze the whole model and instead learn a set of continuous ‘virtual token’ vectors that are prepended to the input or to each layer’s keys and values, steering behavior without touching any weight. All three share LoRA’s core bargain — freeze the giant pretrained model, train a sliver — and differ mainly in where the sliver lives and whether it can be folded back in. LoRA won the popularity contest because of that last property: near-full-FT quality, tiny footprint, and zero-overhead merged inference.
Pitfalls and a decision rule
A few traps recur. Rank too low for a genuinely hard task underfits — the validation loss plateaus high; raise r or widen the injected matrices before blaming the data. Learning rate too high for LoRA still destabilizes, frozen base or not; the safety net is not infinite. Too many epochs on a small set memorizes and hurts generalization in every regime. And do not forget the base is still resident: LoRA slashes the optimizer-state memory, but you must still hold the frozen weights — quantize them (QLoRA) if that is your bottleneck.
The decision rule is short. Reach for full fine-tuning only when you have the hardware, a large high-quality dataset, and a task that genuinely needs deep changes to the model’s knowledge. Reach for LoRA / QLoRA in almost every other case: adapting tone, format, domain, or task on limited compute, or whenever you want many cheap, swappable, non-destructive task variants over one preserved base. For the overwhelming majority of real fine-tuning work, the low-rank update is not a compromise — it is the right default, and the memory ledger is why.