From RLHF to DPO and GRPO: How Preference Training Reshapes a Model

Every instruction-tuned model you deploy has been reshaped by preference training. Supervised fine-tuning teaches it the format of a good answer; preference training then moves probability mass toward the answers people (or a verifier) prefer and away from the ones they reject. The algorithm you choose for that second stage decides how many models sit in GPU memory, how many responses you have to generate, which failure modes you will meet, and what a single hyperparameter called beta actually does.

This guide works through the three families that matter in practice: RLHF with a learned reward model and PPO, Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO). It derives what each objective optimises, does the arithmetic on memory and sampling cost, explains why groups of identical rewards waste compute, and shows how reward hacking appears in metrics before it appears in user complaints. The embedded lab trains a small synthetic policy with DPO or GRPO so you can watch each effect happen.

The shared objective: reward, anchored to a reference

All three methods are approximations of the same objective. Given a prompt x, a policy π and a frozen reference policy πrefpi;_ref (usually the SFT checkpoint), find the policy that maximises expected reward while paying a penalty for drifting away from the reference:

maximise  E[ r(x, y) ]  -  beta * KL( pi(. | x) || pi_ref(. | x) )
          y ~ pi(. | x)

closed-form optimum (for a fixed reward r):
pi*(y | x)  =  pi_ref(y | x) * exp( r(x, y) / beta ) / Z(x)

The closed form is worth reading slowly. The optimal policy is the reference policy re-weighted by an exponential of the reward. Beta is a temperature on that re-weighting: with a large beta the policy stays close to the reference and only nudges toward high-reward answers; with a small beta it concentrates almost all mass on whatever the reward function rates highest. That is the whole reason the KL term exists. The reward is never the true goal, only a proxy for it (a learned reward model, a unit test, a regex for the final answer), and the further the policy moves, the more likely it finds answers where the proxy and the goal disagree.

The KL term also protects capabilities that the preference data never mentions. A policy tuned hard on support-ticket tone can quietly lose arithmetic or code ability, because nothing in the reward measures them. Keeping KL small is a cheap insurance policy against regressions you are not testing for.

RLHF with PPO: four models and a critic

The classic pipeline, popularised by InstructGPT, has three stages. Supervised fine-tuning produces the reference. Labellers compare pairs of answers, and a reward model is trained on those comparisons with a Bradley-Terry loss, -log σ(r(x, y_w) - r(x, y_l)), so the score difference predicts the probability that the chosen answer wins. Finally PPO optimises the policy against the reward model, with a per-token KL penalty to the reference folded into the reward.

PPO needs a value model (the critic) to estimate how good a partial answer is, so it can assign credit token by token. That means four networks are involved at once: the policy and the critic being trained, plus the frozen reference and reward model. For a 7B model in full fine-tuning with Adam in mixed precision, a common rule of thumb is about 16 bytes per trained parameter (weights, gradients, fp32 master copy and two optimizer moments), so roughly 112 GB for the policy and another 112 GB for a critic of the same size, plus about 28 GB for the two frozen bf16 models. That is about 250 GB before activations and before the generation engine's KV cache, which is why PPO-based RLHF became an infrastructure project in its own right. These are calculated estimates; sharding, LoRA and offloading change them a lot.

PPO also has many interacting knobs: clip range, value-loss coefficient, GAE lambda, KL coefficient or adaptive KL target, number of PPO epochs per batch. It works, and it remains the most general option when you have a good reward model and need token-level credit assignment, but it is expensive to run and easy to destabilise.

DPO: preference learning as classification

DPO starts from the closed-form optimum above and inverts it. If the optimal policy is the reference re-weighted by exp(r / beta), then the reward can be written in terms of the policy: r(x, y) = beta * log(π(y|x) / π_ref(y|x)) + beta * log Z(x). Substitute that into the Bradley-Terry loss and the partition function Z(x) cancels, because it is the same for both answers to one prompt. What remains is a loss on log-probabilities only:

def dpo_loss(pi_logp_w, pi_logp_l, ref_logp_w, ref_logp_l, beta=0.1):
    # each argument: summed log-probability of the whole answer, shape [batch]
    margin = (pi_logp_w - ref_logp_w) - (pi_logp_l - ref_logp_l)
    return -torch.nn.functional.logsigmoid(beta * margin).mean()

# implicit reward of an answer, useful for monitoring:
# r_hat(y) = beta * (pi_logp(y) - ref_logp(y))

No reward model, no sampling during training, no critic. You need a fixed dataset of (prompt, chosen, rejected) triples, the policy being trained and the reference (whose log-probabilities can even be precomputed once). Training looks like ordinary supervised learning, which is why DPO spread so quickly.

Beta has a precise meaning here. The gradient on a pair is weighted by σ(-beta * margin), so a pair stops contributing once beta * margin is a few units. With beta = 0.1, the log-ratio margin saturates around 30 to 50 nats; with beta = 0.5 it saturates around 6 to 10. A smaller beta therefore lets each pair drag the policy further from the reference before it is considered learned. That is what you want with clean, plentiful data and exactly what you do not want with noisy labels, because a mislabelled pair is also learned to a large margin.

DPO's weaknesses follow from it being offline. It only learns about answers that appear in the dataset, usually sampled from some other model, so it cannot discover better answers on its own. Both the chosen and rejected log-probabilities often fall during training, with the probability mass going to answers outside the dataset. And because the loss only sees log-probability differences, it inherits any bias in how the pairs were labelled: if longer answers were preferred more often, length becomes part of what the model learns.

GRPO: a group is its own baseline

GRPO, introduced in the DeepSeekMath paper and used at scale for reasoning models, keeps the online sampling of PPO but removes the critic. For each prompt the policy generates a group of G answers. Each answer is scored, and its advantage is its reward standardised within the group:

for each prompt x in the batch:
    y_1 .. y_G  ~  pi_old(. | x)                    # G samples, same prompt
    r_i = reward(x, y_i)                            # verifier, rule or reward model
    A_i = (r_i - mean(r)) / (std(r) + eps)          # same advantage for every token of y_i
loss = - mean_i mean_t [ min(ratio_t * A_i, clip(ratio_t, 1-eps, 1+eps) * A_i) ]
       + beta * KL_estimate(pi || pi_ref)           # KL in the loss, not in the reward

The group mean replaces the value model as a baseline. Answers better than their siblings are reinforced and worse ones are pushed down, so the update is about relative quality on the same prompt, which suits rewards that are only meaningful in comparison. Memory drops to the trained policy plus a frozen reference (and a reward model if you use one), roughly half of PPO in the arithmetic above. The cost moves to generation: every step needs prompts × G fresh completions, so the inference engine (often vLLM inside the training loop) dominates wall-clock time.

GRPO pairs naturally with verifiable rewards: unit tests for code, exact-match for maths, a schema validator for structured output. A verifier cannot be flattered by tone or length, which removes the most common reward-hacking route. It can still be gamed through its own loopholes, such as tests that do not cover the edge case or an answer extractor that accepts a lucky format, so inspect samples rather than trusting the pass rate.

The zero-signal group: why group size is a probability question

Standardising within a group has a sharp edge. If every answer in the group receives the same reward, the standard deviation is zero, every advantage is zero, and the group contributes no gradient at all. You paid for G generations and learned nothing. With a binary verifier and a per-prompt pass rate p, the probability of a dead group is:

P(no signal) = p^G + (1 - p)^G

pass rate p = 0.30           pass rate p = 0.05 (hard prompts)
G = 2   : 0.58               G = 2   : 0.90
G = 4   : 0.25               G = 8   : 0.66
G = 8   : 0.06               G = 16  : 0.44
G = 16  : 0.003              G = 64  : 0.04

Two regimes waste compute. Early on, hard prompts produce all-fail groups; late in training, solved prompts produce all-pass groups. The first is a capability problem (the model rarely finds a correct answer, so a larger G or an easier curriculum is needed); the second is a sign of convergence. Later work such as DAPO addresses this with dynamic sampling, which drops groups whose rewards are all equal and keeps sampling until the batch is full of informative groups. Other variants question the standard-deviation normalisation itself, because it up-weights prompts whose rewards barely vary. Track the fraction of zero-variance groups per step as a first-class training metric.

Method	Models in memory	Samples during training	Signal	Typical failure
PPO (RLHF)	Policy, critic, reference, reward model	Yes, one or more per prompt	Reward model plus per-token KL	Instability, reward-model exploitation
DPO	Policy, reference (or cached log-probs)	No, fixed pairs	Chosen versus rejected	Overfits label noise, limited to dataset answers
GRPO	Policy, reference, optional reward model	Yes, G per prompt	Reward relative to the group	Dead groups, generation cost, proxy hacking with learned rewards

Reward hacking: the proxy rises, the goal does not

Reward models are trained on finite comparisons and carry the biases of their labellers. Length is the best-documented one: across several RLHF datasets, longer answers were preferred often enough that reward models learned to score length, and policy optimisation then learned to produce it. The scaling study of reward-model overoptimisation showed the general shape. As the policy moves away from the reference (measured by KL), the proxy reward keeps rising while the true reward, judged by a stronger "gold" model, rises, peaks and falls.

That gives you the detection recipe. Never monitor only the training reward. Log, per evaluation checkpoint: KL to the reference; mean response length and its distribution; win rate against the reference judged by something other than the training reward (a held-out judge, a different reward model, a human sample); and per-category scores on capability benchmarks the preference data does not cover. The signature of hacking is a training reward that keeps climbing while the independent win rate flattens or falls and length or KL keeps growing.

The fixes, in order of how much they address the cause: fix the signal (a verifier where one exists, a reward model retrained with length-balanced pairs, explicit length normalisation or penalties); stop earlier, choosing the checkpoint by the independent judge rather than the last step; and increase beta to slow the drift. Beta alone rarely saves you, because it slows movement in every direction, including the useful ones.

The knobs in real trainers

Hugging Face TRL exposes all three families, and other frameworks (OpenRLHF, verl, NeMo-Aligner) use similar concepts under different names. The table lists the concepts and the TRL names as an orientation; names and defaults have changed between releases, so check your version's documentation before copying anything.

Concept	TRL name (check your version)	What it controls
KL strength	`beta` in `DPOConfig` and `GRPOConfig`	How far the policy may move from the reference; in recent GRPO versions the default has been changed, so confirm it
Group size	`num_generations`	Samples per prompt, hence dead-group rate and generation cost
Completion length	`max_completion_length`	Generation cost and truncation of long reasoning
Sampling temperature	`temperature`	Diversity within a group; too low means identical answers and no signal
Loss variant	`loss_type`	For DPO, sigmoid versus alternatives such as IPO; for GRPO, how token losses are normalised
Reward scaling	`scale_rewards`	Whether advantages are divided by the group standard deviation

Two practical notes. First, DPO's beta and GRPO's beta share a name but not a role: in DPO it scales the implicit reward inside the loss, in GRPO it weights an explicit KL penalty. Values that are sensible for one say nothing about the other. Second, GRPO's generation step should use the same sampling settings at training and evaluation time, or the measured win rate will not match the training curves.

A worked example: Belgavi's Book Store

Suppose the hypothetical Belgavi's Book Store has two tuning jobs. The first is a model that writes SQL for inventory reports, which can be checked by running the query on a test database and comparing the result. The SFT model passes on about 30% of 4,000 training prompts. The second is the tone of customer replies about returns, which has no verifier, only 6,000 preference pairs from staff reviewers.

For SQL, GRPO with the execution check as the reward is the natural fit. With p = 0.3, a group of 8 is dead about 6% of the time early on, against 58% for a group of 2. A step of 64 prompts with G = 8 at 300 completion tokens generates 64 × 8 × 300 = 153,600 tokens; 300 steps is about 46 million generated tokens. At an illustrative 20,000 tokens per second of generation throughput that is roughly 38 minutes of pure sampling, before the training passes. If the team doubles G to 16 the dead-group rate barely moves (it is already low) while generation cost doubles, so the money is better spent on more prompts or steps. As the pass rate climbs past 90%, all-pass groups appear, and dropping solved prompts from the batch recovers that budget.

For tone, DPO on the staff pairs is cheaper and simpler. A quick audit shows chosen answers are on average 35% longer than rejected ones, so the team rebalances the pairs by length before training, starts at beta = 0.1, and evaluates checkpoints by win rate against the SFT model using a separate judge, with length and KL logged beside it. If around a fifth of the labels look inconsistent on review, raising beta and collecting more pairs per prompt is safer than training longer. All figures here are calculated for illustration.

Common mistakes, and how the lab maps to them

The recurring errors are easy to list. Reporting only the training reward, and discovering hacking from user complaints. Running GRPO with a small group on prompts the model cannot solve, so most of the generation budget produces zero gradient. Sampling with a temperature so low that every answer in a group is identical. Treating DPO's beta as a learning rate. Building DPO pairs where the chosen answer is systematically longer, more formatted or from a stronger model, and then wondering why the tuned model imitates that style. Dropping the KL term entirely without an independent capability check.

The lab reproduces these on a toy problem: 40 prompts with ten candidate answers each, a reference policy, and a true quality score the trainer never sees. Mission one gives GRPO two samples per prompt and asks you to restore the learning signal. Mission two turns on a length-biased reward model and asks for a real win-rate gain without longer answers. Mission three makes the prompts hard and puts you on a rollout budget, so you trade group size against steps. Mission four asks you to run DPO on noisy labels and keep the policy close to the reference while still winning.

Run the lab: DPO vs GRPO: train a toy policy without hacking the reward

The lab trains a tabular policy over ten candidate answers for each of 40 synthetic prompts. On the left, bars show how likely the trained policy is to pick each answer, sorted from best to worst by a hidden true quality score; the dashed line is the reference policy, green dots mark answers the verifier accepts, and the violet strip shows answer length. On the right, the win rate against the reference and the share of groups or pairs that carried no signal are plotted over training steps. Start with the default GRPO setup, then load mission one.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

Add an independent win-rate evaluation (a judge or reward model not used in training) and plot it next to the training reward for every run.
Log KL to the reference, mean completion length and its p95 at every evaluation step, and alert when length grows while the independent win rate does not.
For GRPO runs, record the fraction of zero-variance groups per step and split it into all-fail and all-pass.
Audit your DPO pairs: compare the length, formatting and source model of chosen versus rejected answers before training.
Where a task can be checked automatically, prototype a verifier-based reward and compare it with the reward model on a held-out set.
Look up the beta, group size and loss-normalisation defaults in the exact trainer version you use and write them into the run config explicitly.

Sources and further reading

Related in this series:

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.