Transformers 5.22: Training LoRA Adapters on FP8 Checkpoints, the Backward Pass and the Memory Arithmetic

What changed

Transformers 5.22.0, published on 29 October 2026, says that FP8 quantized models can now be used for adapter training, such as LoRA with PEFT: backward passes were added to the dense and grouped FP8 matmuls, and the quantizer's is_trainable is now true, so Trainer accepts a PEFT model on an FP8 base. The 5.22.1 patch the next day only removed a work-in-progress documentation page. Two pull requests carry the change. PR 50031 adds the backward pass. PR 50117 reduces the memory that backward pass needs for mixture-of-experts layers.

The bug it fixes: gradients that quietly vanished

PR 50031 describes the earlier behaviour precisely. The FP8 matmuls run in kernels with no autograd formula, so FP8Linear returned outputs without a grad_fn, and the gradient to every earlier layer was silently dropped; the grouped experts path raised an error in backward instead. Silently is the important word. Residual connections and the adapters' own branches still carried some gradient, so training ran and the loss could still move, while the gradients reaching earlier layers were wrong, and nothing reported it.

The fix adds an input gradient only. The base weights are frozen, so no weight gradient is needed; the gradient with respect to the layer input is computed through the dequantized weight. On a tiny FP8 test model, the PR reports the input gradient within 4.2 percent of an FP32 dequantized reference with both eager and grouped experts, against 96.6 percent off with eager before the change. FP4 weights, and the batched and DeepGEMM expert backends, have no backward yet.

The memory saving, as reported

Backward through a mixture-of-experts layer has to dequantize expert weights. PR 50117 dequantizes in chunks of at most 1 GiB of experts and skips the autograd path at inference. The reported numbers are the PR author's, on an H200 with DeepSeek-V3 layer shapes, not measurements made for this guide:

The PR also states the gradients were bitwise equal to those of PR 50031 for the layer test, and that a full Qwen3-30B-A3B training step's peak was dominated by activations, so the chunking matters most for the layer's own transient rather than end-to-end peak.

The arithmetic for your own model

Fine-grained FP8 in Transformers stores weights in 8 bits with one scale per 128 by 128 block, according to its documentation, so the scale overhead is one value per 16,384 weights, about 0.02 percent at 4 bytes per scale. For a LoRA run the frozen base therefore costs roughly half of bf16. The adapters and their optimiser state are small. A calculation for a dense model with hidden size 4096 and 32 layers, treating the four attention projections as 4096 by 4096 for simplicity:

Illustrative python, checked for syntax; adapt names and storage to your system.

hidden, layers, rank, targets = 4096, 32, 16, 4
lora_params = layers * targets * rank * (hidden + hidden)   # A is r x d_in, B is d_out x r
adam_bytes = lora_params * (4 + 4 + 8)                       # fp32 weight, grad, two moments
base_params = 8_000_000_000

print(lora_params)                                           # 16777216
print(round(adam_bytes / 2**30, 2))                          # 0.25 GiB for adapters and Adam
print(round(base_params * 2 / 2**30, 1))                     # 14.9 GiB of bf16 base weights
print(round(base_params * (1 + 4 / 16384) / 2**30, 1))       # 7.5 GiB of FP8 base weights

Activations come on top and usually dominate at long sequence lengths, which is why gradient checkpointing still matters. The point of the arithmetic is the ratio: on a fixed GPU, an FP8 base roughly halves the largest term that does not shrink with batch size, and a run that did not fit in bf16 may now fit without moving to 4-bit QLoRA. The documentation also notes the hardware requirement: compute capability 9.0 or newer, such as H100.

Setting up an adapter run on an FP8 base

Illustrative Python, checked for syntax only; module names depend on the architecture.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

base_id = 'your-org/your-fp8-checkpoint'          # a fine-grained FP8 checkpoint
model = AutoModelForCausalLM.from_pretrained(base_id, dtype='auto', device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(base_id)

config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                    target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj'],
                    task_type='CAUSAL_LM')
peft_model = get_peft_model(model, config)
peft_model.print_trainable_parameters()          # only LoRA weights should be trainable

A checkpoint saved with fine-grained FP8 carries its quantization config, so loading it keeps the weights in FP8. The base stays frozen; only the adapter matrices train. Check the module names against the model, since target names differ between architectures, and for mixture-of-experts models check which experts backend is active before training.

Verify gradients against a reference

Because the old failure was silent, do not trust a falling loss. The check that would have caught it is the one the PR used: on a small model, compare input gradients from the FP8 model with those of a copy dequantized to bf16 or FP32, and expect a difference of a few percent, not 96.6 percent. Keep that comparison as the test of record. A per-layer gradient report is a cheap smoke test on top, for adapters that are not training at all:

Illustrative python, checked for syntax; adapt names and storage to your system.

def adapter_grad_report(model):
    """Return {layer_index: max abs grad over LoRA params} after loss.backward()."""
    report = {}
    for name, param in model.named_parameters():
        if 'lora_' not in name or not param.requires_grad:
            continue
        layer = next((int(part) for part in name.split('.') if part.isdigit()), -1)
        g = 0.0 if param.grad is None else float(param.grad.abs().max())
        report[layer] = max(report.get(layer, 0.0), g)
    return dict(sorted(report.items()))

# after one training step:
# report = adapter_grad_report(peft_model)
# assert all(v > 0 for v in report.values()), report

A non-zero report does not prove the gradients are right. The old code produced wrong gradients, not necessarily missing ones, so this smoke test alone would probably not have caught it; it catches frozen adapters and broken target modules. The reference comparison catches wrong values.

Upgrade notes

• The release also lists a breaking change in mixture-of-experts expert-parallel code, which moved to its own file; check imports if you patch EP logic.
• Adapter training through FP4 weights and through the batched or DeepGEMM expert backends still has no backward, according to PR 50031; choose a supported experts implementation for training.
• Pin Transformers and PEFT together and keep the reference gradient comparison as a test, so a later regression is caught by a check rather than by a model that quietly learned less.
• Treat the PR memory figures as one configuration on one GPU; measure peak memory with your sequence length and batch size before planning capacity.

Practice and related reading

Further reading:

• QLoRA: frozen weights and gradient flow
• LoRA and QLoRA trainable parameters
• Parameter-efficient fine-tuning

Sources and scope

Checked against these sources on 1 November 2026. Event: Transformers v5.22.0 release, 29 October 2026; recheck it before acting. Benchmark and accuracy figures are quoted from pull requests 50031 and 50117. The memory figures in the code block are calculations made for this guide; no GPU training run was performed.

• Transformers v5.22.0 release notes (29 October 2026)
• huggingface/transformers pull request 50031: Train adapters on top of FP8 weights
• huggingface/transformers pull request 50117: FP8 backward, chunked expert dequant, skip autograd.Function at inference
• Transformers documentation: Fine-grained FP8 quantization
• PEFT documentation: LoRA developer guide