SLM Fine-Tuning, in depth: choosing between full fine-tuning, adapters, continued pretraining and DPO, with the memory math and training loops for each

First decide whether to fine-tune at all

Fine-tuning changes how a model behaves: its format, its tone, its handling of a narrow task, the vocabulary it is comfortable with. It is a poor way to add facts that change, because facts learned in weights cannot be updated without retraining and cannot be cited. Before training anything, try a strong prompt with examples and, if the problem is knowledge, retrieval. Fine-tune when those plateau and you can state what behaviour should change.

A common and effective pattern is distillation: use a large model to generate or label training data, then fine-tune a small one to imitate it on your task. The distillation guide covers that data pipeline.

The stages: continued pretraining, SFT, preference tuning

Continued pretraining keeps the original objective, predicting the next token of raw text, but on your domain corpus. Every token contributes to the loss. It teaches vocabulary, notation and style of a field, not how to answer questions. Supervised fine-tuning trains on prompt and response pairs, with the loss computed only on the response, and teaches the task. Preference tuning trains on a prompt with a preferred and a dispreferred response and teaches the model to rank them, which shapes behaviour that is hard to write down as a single correct answer.

Each stage is a separate run with its own data, learning rate and evaluation. Most projects need only SFT. Add continued pretraining when the base model visibly struggles with the domain's text, and add preference tuning when SFT outputs are acceptable but you can reliably say which of two answers is better.

Parameter scope and the memory math

The biggest practical difference between full fine-tuning and adapters is memory. For full-parameter training with AdamW in mixed precision, a standard configuration keeps bf16 weights (2 bytes per parameter), bf16 gradients (2 bytes), an fp32 master copy of the weights (4 bytes) and two fp32 Adam moments (4 bytes each). That is 16 bytes per parameter before activations. Setups that train directly in bf16 without a master copy drop to about 12 bytes, at some risk to numerical stability; 8-bit optimizer states shrink the moments further.

Activations come on top and scale with batch size, sequence length and depth. Gradient checkpointing recomputes activations during the backward pass instead of storing them, costing roughly one extra forward pass. Sharding the weights, gradients and optimizer states across GPUs, as FSDP and ZeRO do, divides the static memory by the number of devices. With these tools a 1.5B model is a full fine-tune on a single 40 to 80 GB accelerator; a 3B model usually needs sharding or an 8-bit optimizer.

Memory is not the only difference. Full fine-tuning can change the model more, which helps continued pretraining and large behaviour shifts, and it also forgets more. Adapters produce a small artifact per task that can be swapped at serving time, as described in multi-LoRA serving. A reasonable default is LoRA for SFT and preference tuning, and full-parameter training for continued pretraining when the domain shift is large and the model is small.

A full-parameter training loop

The loop below is plain PyTorch around a Hugging Face-style causal language model that returns a loss when given labels. It includes the pieces that matter at this scale: bf16 autocast, gradient accumulation to reach a useful batch size, gradient clipping, linear warmup into cosine decay, and evaluation at a fixed cadence. Data preparation, including masking prompt tokens out of the loss, is assumed; it is identical to the adapter case.

import math, torch

def lr_at(step, total, warmup, peak):
    if step < warmup:
        return peak * step / warmup
    progress = (step - warmup) / max(1, total - warmup)
    return peak * 0.5 * (1 + math.cos(math.pi * progress))       # cosine to zero

def train_full(model, train_loader, eval_fn, total_steps, peak_lr=1e-5,
               warmup=100, accum=8, clip=1.0, eval_every=200):
    # Batches already carry labels with prompt tokens masked (see the LoRA article).
    opt = torch.optim.AdamW(model.parameters(), lr=peak_lr,
                            betas=(0.9, 0.95), weight_decay=0.1)
    model.gradient_checkpointing_enable()     # trade compute for activation memory
    model.train()
    step, micro = 0, 0
    for batch in train_loader:
        with torch.autocast("cuda", dtype=torch.bfloat16):
            loss = model(**batch).loss / accum
        loss.backward()
        micro += 1
        if micro % accum:
            continue
        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)
        for g in opt.param_groups:
            g["lr"] = lr_at(step, total_steps, warmup, peak_lr)
        opt.step()
        opt.zero_grad(set_to_none=True)
        step += 1
        if step % eval_every == 0:
            print(step, eval_fn(model))       # task metric AND general regression set
            model.train()
        if step >= total_steps:
            break

The peak learning rate is the hyperparameter that most often ruins a full fine-tune. Full-parameter runs typically use rates an order of magnitude lower than LoRA runs, commonly around 1e-5, because every weight moves. Too high, and the model forgets general ability within a few hundred steps; watch the general regression score, not just the training loss. Train for one to three epochs over SFT data; more usually memorises. For the optimizer, checkpoint and resume mechanics at scale, see the pretraining loop article.

Continued pretraining without wrecking the model

Continued pretraining is where catastrophic forgetting bites hardest, because the data distribution shifts entirely. Three practices keep it under control. First, mix in replay data: a share of general text similar to the original pretraining mix, so the model keeps rehearsing what it knew. The right share depends on how far the domain is from general text; tune it by measuring general benchmarks. Second, use a lower peak learning rate than original pretraining, with a short warmup, since the model is already near a good solution. Third, pack documents into full-length sequences separated by end-of-sequence tokens, so no compute is wasted on padding.

Packing has a subtle cost: with standard causal attention, a token can attend to the tail of the previous, unrelated document in the same sequence. For pretraining this is generally tolerated; for short SFT examples it can leak context between examples, so either avoid packing there or use attention masks that reset at document boundaries if your kernel supports them.

Preference tuning with DPO

Direct Preference Optimization, introduced by Rafailov and colleagues in 2023, replaces the reward model and reinforcement learning of classic RLHF with a single classification-style loss. For each prompt it needs a chosen response y_w and a rejected response y_l, plus a frozen reference model, usually the SFT checkpoint. The loss is the negative log-sigmoid of beta times the margin, where the margin is how much the policy's log-probability ratio against the reference is higher for the chosen response than for the rejected one.

import torch
import torch.nn.functional as F

def seq_logprob(model, input_ids, attention_mask, labels):
    """Sum of log-probabilities of the response tokens (labels != -100)."""
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits[:, :-1]
    targets = labels[:, 1:]
    mask = targets != -100
    logp = torch.log_softmax(logits.float(), dim=-1)
    tok = torch.gather(logp, 2, targets.clamp(min=0).unsqueeze(-1)).squeeze(-1)
    return (tok * mask).sum(-1)

def dpo_loss(policy, reference, chosen, rejected, beta=0.1):
    pc = seq_logprob(policy, **chosen)
    pr = seq_logprob(policy, **rejected)
    with torch.no_grad():                      # the reference model never trains
        rc = seq_logprob(reference, **chosen)
        rr = seq_logprob(reference, **rejected)
    margin = (pc - rc) - (pr - rr)             # how much more the policy prefers "chosen"
    loss = -F.logsigmoid(beta * margin).mean()
    return loss, {"margin": margin.mean().item(),
                  "chosen_shift": (pc - rc).mean().item(),
                  "rejected_shift": (pr - rr).mean().item()}

Worked example with beta 0.1. At the start the policy equals the reference, the margin is 0, and the loss is ln 2, about 0.693. If training raises the chosen response's log-ratio by 2.0 and lowers the rejected one by 1.0, the margin is 3.0 and the loss falls to about 0.554. If the model gets the pair backwards by the same amount, the loss is about 0.854. Beta controls how far the policy may drift from the reference: small values allow large moves.

Log the chosen and rejected shifts separately. A frequent failure is that both fall, with the rejected falling faster: the margin grows while the model becomes less likely to produce the chosen answer too. Another is length exploitation, where chosen responses are systematically longer and the model learns verbosity rather than quality. Balance lengths in the data and check output length during evaluation.

Failure modes and how to see them

Operating fine-tunes as a pipeline

Treat a fine-tune as a build, not an experiment. Version the dataset, the base model revision, the code and every hyperparameter together, so any shipped model can be reproduced. Keep three evaluation sets: the task set, a general regression set covering instruction following and reasoning the base model handled, and a safety and format set. A run ships only if the task improves and the others do not fall beyond an agreed tolerance.

Select checkpoints by evaluation score, not by the last step. After selection, merge adapters if used, then quantize for deployment and re-run the evaluation on the quantized artifact, because quantization can erase small fine-tuning gains; the quantization article covers formats. Check the base model's license for fine-tuning and redistribution terms before you start, not before you ship.

What to do next

1. Write down the behaviour you want to change and confirm that prompting and retrieval have plateaued.
2. Build three evaluation sets: task, general regression and safety or format, and score the base model on all three.
3. Choose stages: SFT alone unless you have evidence for continued pretraining or preference data.
4. Compute memory with the 16 bytes per parameter rule and pick full or adapter scope for each stage.
5. Run a short SFT with a conservative learning rate, evaluating every few hundred steps on all three sets.
6. If adding DPO, log chosen and rejected shifts separately and balance response lengths.
7. Quantize the selected checkpoint and re-run the full evaluation before shipping.