Machine Unlearning for LLMs, in depth: forget and retain sets, gradient ascent to NPO and RMU, TOFU, MUSE and WMDP, and why forgotten knowledge comes back

What forgetting should mean

Split the training data into a forget set D_f and everything else. The gold standard is the model you would get by retraining on the data minus D_f; call it the reference. An unlearning method succeeds if the unlearned model is hard to tell apart from the reference, both on D_f (it should not know more than the reference) and elsewhere (it should not know less). Because you usually cannot afford the reference, you approximate it with a retain set D_r, a sample of data the model should keep performing well on, and with held-out evaluations.

Two consequences follow. First, forgetting is not the same as refusing: a model that says "I can't help with that" about a person may still hold their details in its weights. Second, forgetting below the reference is also a failure. If a model forgets everything about a famous author because one fan-fiction file was removed, it is distinguishable from the reference and has lost utility.

Objectives: from gradient ascent to NPO and RMU

Nearly every method combines a term that pushes the model away from D_f with a term that anchors it on D_r. They differ in what "push away" means.

• Gradient ascent. Maximise the language-modelling loss on D_f. Simple and unstable: the loss is unbounded, so the model quickly collapses into gibberish on everything.
• Gradient difference. Gradient ascent on D_f plus ordinary descent on D_r. Slower collapse, but it still overshoots.
• KL on retain. Replace the retain loss with a KL penalty to the original model's outputs on D_r, so retained behaviour stays close to where it started.
• NPO (negative preference optimisation, Zhang et al., 2024). Treat the forget data as dispreferred responses in a DPO-style loss with no preferred side. The loss is (2/beta) times the mean of log(1 + (p_theta(y|x) / p_ref(y|x))^beta). It saturates once the model's likelihood falls well below the reference, which removes gradient ascent's runaway.
• RMU (representation misdirection, from the WMDP paper, Li et al., 2024). Instead of outputs, act on hidden states: at one chosen layer, steer activations on forget text toward a fixed random direction scaled by a constant, and keep activations on retain text close to the frozen model's. Only a few layers are updated.
• Refusal or "I don't know" targets. Fine-tune the model to answer forget prompts with a refusal. Cheap and readable, but by construction it trains a behaviour rather than removing knowledge.

A compact NPO plus retain-KL training step in PyTorch looks like this. The structure is the same for the other objectives; only the forget term changes.

import torch, torch.nn.functional as F

def seq_logprob(model, batch):
    """Sum of token log-probs of the labels; padding labels are -100."""
    logits = model(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"]).logits
    logits, labels = logits[:, :-1], batch["labels"][:, 1:]
    mask = labels != -100
    lp = torch.gather(F.log_softmax(logits, -1), 2, labels.clamp(min=0).unsqueeze(-1)).squeeze(-1)
    return (lp * mask).sum(-1)

def unlearn_step(model, ref, forget_b, retain_b, opt, beta=0.1, lam=1.0):
    with torch.no_grad():
        ref_f = seq_logprob(ref, forget_b)
        ref_r = ref(input_ids=retain_b["input_ids"], attention_mask=retain_b["attention_mask"]).logits
    log_ratio = seq_logprob(model, forget_b) - ref_f
    npo = (2.0 / beta) * F.softplus(beta * log_ratio).mean()   # softplus(z) = log(1 + e^z)
    cur_r = model(input_ids=retain_b["input_ids"], attention_mask=retain_b["attention_mask"]).logits
    kl = F.kl_div(F.log_softmax(cur_r, -1), F.log_softmax(ref_r, -1),
                  log_target=True, reduction="batchmean")
    loss = npo + lam * kl
    opt.zero_grad(); loss.backward(); opt.step()
    return npo.item(), kl.item()

Run it for a fixed, small budget (often a few hundred steps at a low learning rate), evaluate after every chunk of steps, and stop on the first checkpoint that passes the forget gate without breaking the retain gate. Unlearning is an early-stopping problem: train too long and utility collapses.

Measuring forgetting

Three public benchmarks cover the three goals in the table above.

• TOFU (Maini et al., 2024) fine-tunes a model on question-answer pairs about 200 fictitious authors, then asks it to forget a fraction of them. Because the authors never existed, the retrained reference is cheap to build, and forget quality is measured by how closely the unlearned model's statistics match it. Model utility is measured on the remaining authors and on real-world facts.
• MUSE (Shi et al., 2024) scores six properties: no verbatim memorisation, no knowledge memorisation, no privacy leakage, utility preservation, scalability with the size of removal requests, and sustainability across sequential requests. Its corpora are news articles and books.
• WMDP (Li et al., 2024) measures hazardous knowledge with multiple-choice questions in biosecurity, chemical security and cybersecurity; the public dataset has 3,668 questions. Pair it with neighbouring MMLU subjects as a retain set, as described in MMLU plus safety subsets.

Benchmarks tell you about a method. For your own deletion you also need a membership inference check: compare the model's loss on D_f with its loss on similar data it never saw. If D_f still has distinctly lower loss, the model still remembers it. The leakage tests in PII leakage in LLMs and the formal guarantees in DP-SGD are the natural companions here.

import numpy as np
from sklearn.metrics import roc_auc_score

def mia_auc(per_example_loss, forget_set, lookalike_set):
    """AUC of 'lower loss means member'. About 0.5 means D_f looks like unseen data."""
    lf = np.array([per_example_loss(x) for x in forget_set])
    lu = np.array([per_example_loss(x) for x in lookalike_set])
    labels = np.r_[np.ones(len(lf)), np.zeros(len(lu))]
    return roc_auc_score(labels, -np.r_[lf, lu])

The look-alike set matters more than the code: draw it from the same distribution as D_f (same ticket type, same time period) but from records never trained on. An AUC of 0.5 is the target; well above 0.5 means the model still separates the forgotten records from strangers, and well below means it has over-forgotten them, which is also detectable.

Why forgotten knowledge comes back

The uncomfortable result of the last two years of research is that low forget-set scores often mean the knowledge has been made hard to reach, not removed. Four recovery paths recur in the literature.

• Relearning. Fine-tuning the unlearned model on a small amount of related, even public, text brings forget-set performance back far faster than learning it from scratch would. Anyone who can fine-tune your open weights can do this.
• Quantization. Unlearning updates are small. A 2024 study published at ICLR 2025 ("Catastrophic Failure of LLM Unlearning via Quantization") reported that, for methods with utility constraints, models retained on average 21 percent of the targeted knowledge at full precision and 83 percent after 4-bit quantization, because rounding snaps weights back toward their original values.
• Rephrasing. Forget evaluations often reuse the training phrasing. Paraphrases, translations, or multi-hop questions that need the fact as an intermediate step recover it.
• Probing and jailbreaks. Linear probes on hidden states, or adversarial prompts, extract information that the output layer no longer volunteers.

The practical rule: never report approximate unlearning as deletion unless the candidate survives a relearning test, evaluation at the precision you ship, paraphrased questions, and a probe. No current method is reliably robust against an attacker who can fine-tune the weights.

Worked example: retrain or unlearn?

A company fine-tuned an open 8B model with a LoRA adapter on 50,000 support tickets. A customer requests deletion; 310 tickets mention them. What should the team do?

1. Locate the data. The base model never saw these tickets; only the adapter did. Scope shrinks from "the model" to "one adapter".
2. Price exact retraining. Re-running the LoRA fine-tune on 49,690 tickets took about two GPU-hours last time. Exact retraining is cheap, auditable and needs no argument about whether forgetting worked.
3. Decide. Retrain the adapter. Approximate unlearning would save two GPU-hours and cost an evaluation campaign plus a weaker claim.
4. Close the other copies. Remove the tickets from the training store, the RAG index, eval sets and caches, and add the customer's identifiers to the ingestion blocklist so the next export does not reintroduce them.
5. Verify. Run a membership-inference comparison and targeted extraction prompts (the customer's name plus order details) against the new adapter.

Approximate unlearning earns its keep in a different situation: knowledge in the pretrained weights, where retraining costs millions. Suppose the same team must reduce hazardous chemistry knowledge in a model before release. They run RMU, gate on WMDP-chem accuracy near the 25 percent chance level for four-option questions while neighbouring MMLU chemistry stays within two points, then add the relearning and 4-bit tests. If the model regains most of the knowledge after a few hundred fine-tuning steps on public papers, they document that the mitigation holds only for API deployment, where users cannot fine-tune, and do not release the weights on that basis.

Failure modes

Trade-offs

Exact retraining is the only method with a clean guarantee, and it is affordable for fine-tunes, adapters and small models; design pipelines so that retraining is a button, not a project. Sharded training makes exact deletion cheaper at the cost of quality and complexity. Approximate unlearning is fast but its guarantee is empirical and attackable, which is acceptable for reducing risk in an API-only model and weak for an open-weights release or a legal deletion claim. Output filters are cheapest and shallowest. Mixing them is normal: retrain what you can, unlearn what you cannot, filter as a last layer, and say clearly which one you did. Unlearning is also no defence against data poisoning you have not found yet; detection comes first.

What to do next

1. Inventory where each data source lands: base model, adapters, RAG indexes, eval sets, caches.
2. Make adapter and fine-tune retraining reproducible and scripted, so exact deletion is the default for anything you trained yourself.
3. Build a forget and retain evaluation harness, including membership inference against unseen look-alike data.
4. Add the four recovery tests to every unlearning gate: relearning, shipped precision, paraphrase and probing.
5. Start experiments with NPO plus retain KL, or RMU for domain-level hazards, with a small step budget and early stopping.
6. Write down which method was used for each request and what it does and does not guarantee.