Evidence label: deterministic synthetic model. The lab is a JavaScript model of LoRA training. Each of seven projection matrices has a hidden target update made of 64 directions with decaying strength, plus 64 noise directions that appear in the training data only. A rank-r adapter on a targeted matrix learns the r strongest directions it sees, each trained as a product of two scalars with Adam and the alpha over r (or root r) scale. Nothing is trained on a real model; every number is a calculated estimate.
Play the lab
How to play
Pick the target modules and rank, then adjust alpha, the learning rate and the scaling mode. Every change re-runs 240 training steps with the same random seed. Watch the tiles: outlines that never fill are directions the adapter cannot reach, amber bars are overshooting, and red bars are memorised noise. The tiles below the canvas show held-out quality, noise memorised, trainable parameters for a 7B-shaped model, adapter plus optimizer memory, and how many steps it took to reach 90% of the final quality.
Missions unlock in order and reward the cheapest configuration that meets each target. The events change the task: a new-domain task whose update lives mostly in the MLP, and a small noisy dataset in which noise directions compete with real ones.
What the model assumes
Each matrix's required update is a set of independent directions, so the learning problem splits into one small bilinear problem per direction: the adapter's contribution is the scale times a times b, with a started at a random value and b at zero, exactly like LoRA's A and B. Adam with standard moments and a little minibatch noise updates a and b. A rank-r adapter is assumed to pick the r strongest directions present in the training data, which is what the best rank-r fit would do. Held-out quality is the share of the true update's energy that the learned update recovers, minus any energy put into noise directions.
What it cannot tell you
Real directions are not independent or perfectly aligned with the adapter, real tasks do not come with a known spectrum, and real training includes warm-up, schedules, dropout, tokenisation effects and interactions between layers. Parameter and memory figures use Llama-2-7B shapes without grouped-query attention. Use the lab to build intuition about where the update must live and how scale and rank interact, then confirm with held-out evaluations on your own model and data.
Read the full explanation
- LoRA Rank, Alpha and Target Modules: Choosing Settings That Actually Work (companion article)
- From RLHF to DPO and GRPO: How Preference Training Reshapes a Model
- Why INT4 Breaks Some Layers: Outliers, Group Size and Activation-Aware Scaling
All AI in Production labs · Lab catalog
- LoRA: Low-Rank Adaptation of Large Language Models
- QLoRA: Efficient Finetuning of Quantized LLMs
- A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (rsLoRA)
- LoRA Learns Less and Forgets Less
- Hugging Face PEFT documentation: LoRA
- vLLM documentation: LoRA adapters
Views are my own and do not represent any employer, past or present.