You cannot write a loss function for ‘be helpful.’ There is no labeled target, no ground-truth token, no differentiable score for a good answer. What you can collect is a comparison: shown two responses to the same prompt, a human can say which one is better. RLHF and DPO are two ways to turn that thin signal — a pile of pairwise preferences — into a change in the model’s weights. RLHF does it in three stages: fit a reward model to the preferences with the Bradley-Terry likelihood, then use reinforcement learning (PPO) to push the policy toward high reward while a KL penalty tethers it to the original model. DPO does it in one stage, and the reason it can is a single beautiful fact: the KL-constrained reward-maximization problem has a closed-form optimal policy, and you can invert that formula to express reward in terms of the policy itself. Substitute it back into the preference likelihood and the reward model — and its intractable normalizer — simply cancels out. This piece walks the whole derivation, gives the exact losses and gradients, and shows why both methods are really optimizing the same objective from opposite ends.
Why alignment needs a preference objective
Pretraining and supervised fine-tuning both optimize the same thing: next-token likelihood on a fixed corpus. That works when there is a target token. Alignment breaks that assumption. ‘Which of these two summaries is more faithful?’ has no single correct string — there are many good answers and many bad ones, and the difference is a matter of degree that no one can spell out as a per-token label.
The trick RLHF uses is to never ask a human to score, only to compare. Absolute quality ratings are noisy and drift between annotators; relative judgments (‘A is better than B’) are far more consistent. So the raw material of alignment is a dataset of triples (x, y_w, y_l): a prompt x, a preferred (winning) response y_w, and a dispreferred (losing) response y_l. The whole game is: given only orderings of pairs, change a language model so it generates more things like y_w and fewer like y_l — without a single explicit target to regress toward. To do that we first need a probabilistic model of what a preference means, and that is where Bradley-Terry enters.
The Bradley-Terry model of preferences
Bradley-Terry is a 1952 model for paired comparisons: each item i has a latent scalar strength s_i, and the probability that i beats j is s_i / (s_i + s_j). In RLHF we make the strength the exponential of a real-valued reward r(x, y), so strengths are positive and rewards live on the whole real line. Writing r_w = r(x, y_w) and r_l = r(x, y_l):
P(y_w > y_l | x)
= exp(r_w) / ( exp(r_w) + exp(r_l) )
= 1 / ( 1 + exp(-(r_w - r_l)) )
= σ( r_w - r_l )The algebra collapses to a logistic (sigmoid) function of the reward difference. This is the load-bearing fact of the whole subject. It says a preference depends only on how much one reward exceeds the other, not on the absolute levels — add a constant to every reward for a given prompt and the predicted preferences are unchanged. That shift-invariance is exactly why reward is only ever identifiable up to a prompt-dependent constant, a loose end that will matter enormously when we get to DPO.
Learning the reward model from preferences
Bradley-Terry gives us a likelihood, so fitting a reward model is just maximum likelihood. Parameterize r_φ(x, y) as a neural network — in practice the pretrained transformer with its language-model head replaced by a single scalar output read off the final token. Then minimize the negative log-likelihood of the observed preferences:
L_R(φ) = − E_(x, y_w, y_l) ~ D [ log σ( r_φ(x, y_w) − r_φ(x, y_l) ) ]Read it as a binary-classification loss: the ‘label’ is always ‘the winner won,’ and the logit is the reward gap. The gradient pushes r_φ(x, y_w) up and r_φ(x, y_l) down, weighted by σ(r_l − r_w) — the model’s current probability of getting the pair wrong. Pairs it already orders confidently contribute almost no gradient; contested pairs dominate. The output is a scalar function that scores any (prompt, response) pair. Crucially it is trained only to reproduce the ordering of the human-labeled pairs; its absolute magnitude and its behavior on responses unlike those in D are unconstrained — a fragility that becomes the reward-hacking problem in the next stage.
The RL fine-tuning objective
With a reward in hand, the second RLHF stage optimizes the language model — now called the policy π_θ(y | x) — to produce high-reward responses. But maximizing reward alone is a trap: the reward model is only accurate near the responses it was trained on, so an unconstrained optimizer will happily march off into gibberish that the reward model mistakenly loves. The fix is to anchor the policy to a reference — usually the supervised-fine-tuned model π_ref — with a KL-divergence penalty:
max_π E_(x ~ D, y ~ π(·|x)) [ r(x, y) ]
− β · KL( π(y|x) || π_ref(y|x) )The first term wants reward; the second term is a leash. β sets the leash length: large β keeps the policy close to π_ref (conservative, low reward), small β lets it roam (higher measured reward, higher risk of exploiting the reward model). This single objective — maximize reward subject to a KL budget from the reference — is the mathematical heart of RLHF, and, as we will see, DPO optimizes exactly the same objective without ever instantiating r.
Why the KL penalty is not optional
It is tempting to see the KL term as a mere regularizer, a bit of hygiene. It is structural. Recall that the reward model was fit only on the distribution of responses that appeared in the preference data. Off that manifold it extrapolates blindly, and language models are ferocious optimizers — given millions of samples they will find the adversarial corners where r_φ is high for the wrong reasons: degenerate repetition, keyword stuffing, sycophantic boilerplate. This is reward hacking, and without a constraint it is the default outcome, not an edge case.
The KL penalty bounds how far the policy can move from a distribution the reward model actually understands. It keeps generations fluent (because π_ref is fluent) and keeps the policy inside the region where the reward signal is trustworthy. There is a genuine tension here: too much KL and you barely improve; too little and you drift into hacked, high-reward-low-quality text. In practice the KL is also monitored during training as an early-warning gauge — a KL that shoots up is the canonical sign the policy has found an exploit. Hold on to this leash idea; it is the thing DPO folds directly into its loss.
PPO in one section
How do you actually maximize an objective whose ‘loss’ comes from sampling text and scoring it with a reward model? You cannot backpropagate through the sampling of discrete tokens, so RLHF reaches for policy-gradient RL, specifically Proximal Policy Optimization (PPO). The loop: sample responses from the current policy, score each with the reward model, fold in the per-token KL penalty to form a shaped reward, estimate advantages (how much better a token was than a learned value baseline expected), and take a gradient step that raises the probability of above-average tokens.
PPO’s signature is the clipped surrogate objective. Let ratio = π_θ(a|s) / π_old(a|s) be the probability ratio between the updated and sampling policies. PPO maximizes min( ratio · A, clip(ratio, 1−ε, 1+ε) · A ), which refuses to reward moving the ratio far outside [1−ε, 1+ε]. That clipping keeps each update small and stable. It works, but it is heavy machinery: you must hold four models in play — policy, reference, reward, and value — run online generation every step, and babysit a famously finicky training loop. That operational weight is precisely what motivates DPO.
The closed-form optimal policy
Here is the pivot on which everything turns. The KL-constrained objective is not just tractable in principle — it has an exact closed-form solution. For any fixed reward r, the policy that maximizes expected reward minus β-scaled KL from π_ref is:
π*(y | x) = (1 / Z(x)) · π_ref(y | x) · exp( r(x, y) / β )
where Z(x) = Σ_y π_ref(y | x) · exp( r(x, y) / β )Intuitively this is the reference policy reweighted by an exponential tilt toward reward. A response keeps its prior probability under π_ref but is multiplied by exp(r/β): high-reward responses get boosted, low-reward ones suppressed, and β controls how sharp the tilt is (small β = aggressive reweighting). The term Z(x), the partition function, just renormalizes so the probabilities sum to one. This is the Gibbs/Boltzmann distribution in disguise, and it is the single most important formula in the DPO story.
Deriving the optimal policy
The closed form is worth deriving, because the derivation shows it is exact, not an approximation. Start from the objective and rewrite the KL explicitly, folding the reward inside a single expectation:
max_π E_y~π [ r(x,y) − β log( π(y|x) / π_ref(y|x) ) ]
= −β · min_π E_y~π [ log( π(y|x) / π_ref(y|x) ) − (1/β) r(x,y) ]
= −β · min_π E_y~π [ log( π(y|x) / ( π_ref(y|x) exp(r(x,y)/β) ) ) ]Now divide and multiply inside the log by Z(x) defined above. The reference-times-exp term becomes exactly Z(x) · π*(y|x), so the bracket turns into log( π(y|x) / π*(y|x) ) − log Z(x). Since Z(x) does not depend on π, the problem reduces to min_π KL( π(y|x) || π*(y|x) ). A KL divergence is minimized — and equals zero — exactly when the two distributions are equal. Therefore the optimum is π = π*, the Boltzmann form above. No gradient descent, no approximation: the constrained problem is solved in closed form.
The partition function problem
If we have π* in closed form, why not just compute it and sample from it? Because of Z(x). The partition function sums π_ref(y|x) exp(r(x,y)/β) over every possible response y — that is every string the model could ever emit, an astronomically large (effectively infinite) set. There is no way to enumerate it and no way to evaluate the normalizer exactly.
This is the same wall that makes energy-based models hard: you can write the unnormalized density trivially, but normalizing it is intractable. It is the reason classical RLHF does not use the closed form and instead runs PPO — policy gradients cleverly sidestep Z(x) because the gradient of the log-policy does not require the normalizer’s value. DPO’s contribution is to sidestep Z(x) a different, cheaper way: not by avoiding it during optimization, but by arranging the algebra so the intractable term cancels analytically before you ever compute anything. That cancellation is the whole trick, and it comes from reading the closed form backwards.
DPO’s key move: solve for the reward
Every derivation so far treated r as known and solved for the optimal π. DPO reverses the arrow. Take the closed-form relation and solve it for the reward instead. From π*(y|x) = (1/Z(x)) π_ref(y|x) exp(r(x,y)/β), take logs and rearrange:
r(x, y) = β · log( π*(y|x) / π_ref(y|x) ) + β · log Z(x)This says something profound: any reward function can be represented by its own optimal policy. The reward is (up to the prompt-dependent constant β log Z(x)) just β times the log-ratio between the aligned policy and the reference. Instead of training a separate reward network and then finding its optimal policy, we can parameterize the reward implicitly through the policy we actually want. Replace π* with our trainable π_θ and define the implicit reward r̂_θ(x, y) = β log( π_θ(y|x) / π_ref(y|x) ). The policy has become its own reward model. All that remains is to fit it to the preference data — and to watch Z(x) disappear.
Substituting into Bradley-Terry: Z cancels
Recall the Bradley-Terry likelihood depends only on the difference of rewards, r(x, y_w) − r(x, y_l). Plug in the expression for reward we just derived, for both the winner and the loser:
r(x,y_w) − r(x,y_l)
= [ β log(π_θ(y_w|x)/π_ref(y_w|x)) + β log Z(x) ]
− [ β log(π_θ(y_l|x)/π_ref(y_l|x)) + β log Z(x) ]
= β log(π_θ(y_w|x)/π_ref(y_w|x))
− β log(π_θ(y_l|x)/π_ref(y_l|x))Both responses share the same prompt x, so they share the same Z(x) — and the two β log Z(x) terms subtract to zero. The intractable partition function is gone, exactly, with no approximation. This is the payoff foreshadowed back at Bradley-Terry: because preferences are shift-invariant in reward, the one quantity we could never compute never needed computing. What remains is a preference probability written entirely in terms of the policy we are training and a frozen reference — both of which we can evaluate exactly for any given sequence.
The DPO loss
Assemble the pieces. The probability the model assigns to the human’s preference is σ of that reward difference, now expressed in log-ratios. Maximizing the likelihood of the observed preferences means minimizing its negative log — the DPO loss:
L_DPO(θ) = − E_(x, y_w, y_l) ~ D [
log σ( β log( π_θ(y_w|x) / π_ref(y_w|x) )
− β log( π_θ(y_l|x) / π_ref(y_l|x) ) ) ]Stare at it, because this is the entire method. There is no reward model, no sampling, no RL loop — just a supervised loss over the static preference dataset. To evaluate it you run four forward passes per example (the policy and the frozen reference, on y_w and y_l), read off the sequence log-probabilities, form two log-ratios, take their β-scaled difference, push it through log-sigmoid, and backpropagate. It trains like ordinary fine-tuning — stable, no online generation, one model to update — yet it is provably optimizing the same KL-constrained reward objective PPO targets. That equivalence, not mere convenience, is why DPO landed so hard.
The gradient and its intuition
The loss is elegant, but its gradient is where the intuition lives. Differentiating and writing the implicit reward r̂_θ(x,y) = β log(π_θ(y|x)/π_ref(y|x)):
∇_θ L_DPO = −β · E [ σ( r̂_θ(x,y_l) − r̂_θ(x,y_w) ) ·
( ∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x) ) ]Three things happen at once. The gradient raises the log-prob of the winner y_w and lowers that of the loser y_l — the direction is exactly what you would hope. And the whole update is scaled by σ(r̂_θ(x,y_l) − r̂_θ(x,y_w)), the probability the implicit reward currently has the pair ranked backwards. Examples the model already orders correctly (winner’s implicit reward well above the loser’s) contribute almost nothing; examples it gets wrong dominate the batch. This adaptive weighting is not a heuristic bolted on — it falls straight out of the log-sigmoid, and it is what prevents the model from wasting capacity on pairs it has already learned or degenerating when the reference and policy agree.
A worked numeric example
Make it concrete with one preference pair and β = 0.1. Suppose under the current policy and the frozen reference the sequence log-probabilities are:
winner y_w: log π_θ = −18.0 , log π_ref = −20.0
loser y_l: log π_θ = −15.0 , log π_ref = −16.0
implicit reward r̂_w = β(−18.0 − (−20.0)) = 0.1 × 2.0 = 0.20
implicit reward r̂_l = β(−15.0 − (−16.0)) = 0.1 × 1.0 = 0.10
margin r̂_w − r̂_l = 0.10
loss = −log σ(0.10) = −log(0.5250) ≈ 0.644The model already prefers the winner in the log-ratio sense (positive margin), so the loss is below log 2 ≈ 0.693 (the value at zero margin) but far from zero — there is still gradient to be had. The update weight is σ(r̂_l − r̂_w) = σ(−0.10) ≈ 0.475: a substantial push. Note the winner is still the less likely sequence in absolute terms (−18 vs −15); DPO does not care about absolute likelihood, only about the margin between policy and reference log-ratios. Widen that margin — make the policy prefer the winner more than the reference does, relative to the loser — and the loss falls.
Beta, the reference, and practical footguns
Two design choices dominate DPO in practice. β is the same KL leash as in RLHF: small β (e.g. 0.1) permits large deviations from the reference and stronger preference-fitting; large β keeps the policy conservative. Push β too low and the log-ratios can blow up, driving the policy far from π_ref and degrading fluency — the same reward-hacking risk, in a new costume. The reference policy matters just as much: π_ref must be the model you actually started from (typically the SFT checkpoint), and it is frozen. Its log-probs appear in every term; a mismatched reference silently changes what the loss means.
A subtler footgun: DPO only ever sees the fixed pairs in D, never fresh samples from the current policy. It can lower the loser’s probability so aggressively that it drags the winner’s probability down too — both absolute likelihoods fall while the margin widens, a documented failure where the model becomes oddly less confident in the very responses it should favor. Monitoring the absolute chosen log-probabilities, not just the loss, is the standard guard.
DPO versus PPO: same objective, different path
The clean way to hold both methods in your head: they optimize the identical KL-constrained objective, and DPO is what you get when you solve the reward-model and RL stages simultaneously in closed form instead of sequentially by sampling. RLHF/PPO keeps an explicit reward model, which lets it generalize — score novel on-policy samples generated during training — and it optimizes online, exploring responses the static dataset never contained. That flexibility costs four models, an unstable loop, and heavy compute.
DPO trades the explicit reward and the online exploration for a stable supervised loss over a fixed dataset: one model to train, no sampling, no value network, reproducible runs. The cost is that it is offline — it can only reweight the preferences it was given and cannot discover high-reward behaviors outside that distribution, and it is more sensitive to how well the preference data covers the space. Neither dominates universally: DPO’s simplicity made it the default for most open alignment work, while PPO-style online RL retains an edge when you can afford it and need the policy to explore. But the math says they are two projections of one objective, and that is the durable idea to carry away.