Start with the update, not the acronym
Full fine-tuning changes the original model parameters. LoRA instead adds trainable low-rank matrices while keeping the original weights frozen. The adapted computation combines the original transformation with an additional learned update. This changes which parameters receive gradients and optimizer state; it does not remove the need to run the underlying model.
For a single dense layer with input width d and output width k, a full weight matrix has d times k parameters. A rank-r adapter commonly introduces two factors containing r times d plus k times r parameters. That arithmetic explains the reduction in trainable parameters. It does not by itself predict training memory, training duration or the quality of the learned behavior.
Frozen does not mean unused
The forward pass still uses the base model, and gradients must propagate through the computation needed to update the adapter. Activations, temporary buffers and adapter optimizer state still consume memory. Longer sequences and larger microbatches can increase the training budget even when the number of trainable parameters stays fixed. Distinguish parameter storage from the complete training allocation.
The adapter target list matters too. Training adapters on one projection is a different configuration from attaching them to many projections in every layer. Rank, target modules, scaling and dropout belong in the experiment record. A small adapter file is not evidence that the training process used equally little memory or that every base-model architecture behaves identically.
Lab: change the adapter rank
This browser calculator models one hypothetical square dense layer of width 1,024. Change the adapter rank and inspect full-matrix parameters, adapter parameters and the ratio between them. It performs arithmetic only: there is no neural network, optimizer, dataset or GPU execution behind the control. The expected output is an exact parameter count for the stated factor shapes.
At rank eight, the full matrix contains 1,048,576 parameters and the two adapter factors contain 16,384, or 1.5625 percent of the full matrix count. Doubling rank doubles that adapter count. These are calculated examples, not a measured whole-model training percentage. A real model has multiple layer types, and selected modules determine the total.
Educational calculation on synthetic inputs. No device access, model calls or measured GPU results.
What QLoRA adds
QLoRA combines a quantized frozen base model with trainable adapters. The original paper describes four-bit NormalFloat storage, double quantization and paged optimizers as part of its memory-saving approach. It is a specific training approach, not a label for every model that happens to have four-bit weights and a small auxiliary file.
The quantized base still needs a supported computation path. Model architecture, quantization backend and library versions influence whether a configuration can run. A native-Windows adapter experiment using a higher-precision small model should be described as LoRA, not QLoRA. Likewise, inference through Ollama is not a vLLM experiment. Name the runtime and method that actually produced each result.
Check learning without hiding failures
Before training, generate a synthetic task with deterministic labels. Use different templates and values for training and held-out evaluation. Record a baseline using the same instructions and decoding settings as the adapted model. After training, compare task accuracy, valid-output rate and failure categories. A lower training loss alone does not show useful generalization.
Keep the original checkpoint frozen and test that intended adapter parameters are trainable. Record the number of steps, effective batch size, sequence limit, seed and peak allocator memory. Stop other model runtimes before training to reduce contention, then unload training allocations before inference measurements. Publish aggregate results even when the adapter fails to improve the selected task; that is an informative boundary, provided the method and limitations are clear.
Sources and next steps
vLLM quantization compatibility · PEFT LoRA explanation · QLoRA paper
Follow the Local LLM learning path. Evidence label: explanation and calculated simulation. All inputs are synthetic; no measured training or inference claim is made.