Evidence label: deterministic synthetic model. The lab is a small JavaScript model of preference training. Each of 40 synthetic prompts has ten fixed candidate answers with a hidden true quality, a length, and a reward-model score that includes a small error. GRPO samples groups from the current policy and applies group-standardised advantages with a KL penalty; DPO samples pairs once from the reference, labels them and applies the DPO update. No language model is trained or run, and every number is a calculated estimate.
Play the lab
How to play
Choose a training method and a preference signal, then adjust the group size, beta and number of steps. Every change re-runs training from scratch with the same random seed, so the difference you see comes from your change. The bars animate through training: watch probability mass move from the dashed reference toward the best answers, or toward the longest ones when the reward is biased. The tiles compare the result with your previous setup.
Missions unlock in order. Load the mission setup, fix it, and earn more stars by using fewer sampled responses. The events change the world rather than the trainer: a reward model that likes length, preference labels that are partly flipped, and prompts where only one answer in ten passes.
What the model assumes
The policy is a softmax over ten logits per prompt, so it can represent any distribution over the candidates. GRPO uses eight prompts per step, standardises rewards within each group, and adds the gradient of the exact KL to the reference weighted by beta. DPO forms all pairs among the samples drawn for each prompt and uses an update whose size does not depend on beta, as adaptive optimisers behave in practice, while beta sets where the margin saturates. The win rate compares a sample from the trained policy with a sample from the reference using the hidden quality score; ties count as half.
What it cannot tell you
Real answers are not drawn from a fixed list of ten: a language model generalises across prompts, can invent new answers, and shares parameters between them, so improvements and regressions spread in ways this model cannot show. There is no token-level credit assignment, no clipping, no learning-rate schedule and no capability benchmark. Use the lab to build intuition about signal, drift and proxy rewards, and validate settings on your own model with an independent evaluation.
Read the full explanation
- From RLHF to DPO and GRPO: How Preference Training Reshapes a Model (companion article)
- LoRA Rank, Alpha and Target Modules: Choosing Settings That Actually Work
- Reasoning Models and Thinking Budgets: When Extra Tokens Buy Accuracy
All AI in Production labs · Lab catalog
- Training language models to follow instructions with human feedback (InstructGPT)
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO)
- Scaling Laws for Reward Model Overoptimization
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Hugging Face TRL documentation
Views are my own and do not represent any employer, past or present.