Why alignment needs a preference objective
Pretraining and supervised fine-tuning both optimize the same thing: next-token likelihood on a fixed corpus. That works when there is a target token. Alignment breaks that assumption. ‘Which of these two summaries is more faithful?’ has no single correct string — there are many good answers and many bad ones, and the difference is a matter of degree that no one can spell out as a per-token label.
The trick RLHF uses is to never ask a human to score, only to compare. Absolute quality ratings are noisy and drift between annotators; relative judgments (‘A is better than B’) are far more consistent. So the raw material of alignment is a dataset of triples (x, y_w, y_l): a prompt x, a preferred (winning) response y_w, and a dispreferred (losing) response y_l. The whole game is: given only orderings of pairs, change a language model so it generates more things like y_w and fewer like y_l — without a single explicit target to regress toward. To do that we first need a probabilistic model of what a preference means, and that is where Bradley-Terry enters.
The Bradley-Terry model of preferences
Bradley-Terry is a 1952 model for paired comparisons: each item i has a latent scalar strength s_i, and the probability that i beats j is s_i / (s_i + s_j). In RLHF we make the strength the exponential of a real-valued reward r(x, y), so strengths are positive and rewards live on the whole real line. Writing r_w = r(x, y_w) and r_l = r(x, y_l):
P(y_w > y_l | x)
= exp(r_w) / ( exp(r_w) + exp(r_l) )
= 1 / ( 1 + exp(-(r_w - r_l)) )
= σ( r_w - r_l )The algebra collapses to a logistic (sigmoid) function of the reward difference. This is the load-bearing fact of the whole subject. It says a preference depends only on how much one reward exceeds the other, not on the absolute levels — add a constant to every reward for a given prompt and the predicted preferences are unchanged. That shift-invariance is exactly why reward is only ever identifiable up to a prompt-dependent constant, a loose end that will matter enormously when we get to DPO.
Learning the reward model from preferences
Bradley-Terry gives us a likelihood, so fitting a reward model is just maximum likelihood. Parameterize r_φ(x, y) as a neural network — in practice the pretrained transformer with its language-model head replaced by a single scalar output read off the final token. Then minimize the negative log-likelihood of the observed preferences:
L_R(φ) = − E_(x, y_w, y_l) ~ D [ log σ( r_φ(x, y_w) − r_φ(x, y_l) ) ]Read it as a binary-classification loss: the ‘label’ is always ‘the winner won,’ and the logit is the reward gap. The gradient pushes r_φ(x, y_w) up and r_φ(x, y_l) down, weighted by σ(r_l − r_w) — the model’s current probability of getting the pair wrong. Pairs it already orders confidently contribute almost no gradient; contested pairs dominate. The output is a scalar function that scores any (prompt, response) pair. Crucially it is trained only to reproduce the ordering of the human-labeled pairs; its absolute magnitude and its behavior on responses unlike those in D are unconstrained — a fragility that becomes the reward-hacking problem in the next stage.
The RL fine-tuning objective
With a reward in hand, the second RLHF stage optimizes the language model — now called the policy π_θ(y | x) — to produce high-reward responses. But maximizing reward alone is a trap: the reward model is only accurate near the responses it was trained on, so an unconstrained optimizer will happily march off into gibberish that the reward model mistakenly loves. The fix is to anchor the policy to a reference — usually the supervised-fine-tuned model π_ref — with a KL-divergence penalty:
max_π E_(x ~ D, y ~ π(·|x)) [ r(x, y) ]
− β · KL( π(y|x) || π_ref(y|x) )The first term wants reward; the second term is a leash. β sets the leash length: large β keeps the policy close to π_ref (conservative, low reward), small β lets it roam (higher measured reward, higher risk of exploiting the reward model). This single objective — maximize reward subject to a KL budget from the reference — is the mathematical heart of RLHF, and, as we will see, DPO optimizes exactly the same objective without ever instantiating r.
Why the KL penalty is not optional
It is tempting to see the KL term as a mere regularizer, a bit of hygiene. It is structural. Recall that the reward model was fit only on the distribution of responses that appeared in the preference data. Off that manifold it extrapolates blindly, and language models are ferocious optimizers — given millions of samples they will find the adversarial corners where r_φ is high for the wrong reasons: degenerate repetition, keyword stuffing, sycophantic boilerplate. This is reward hacking, and without a constraint it is the default outcome, not an edge case.
The KL penalty bounds how far the policy can move from a distribution the reward model actually understands. It keeps generations fluent (because π_ref is fluent) and keeps the policy inside the region where the reward signal is trustworthy. There is a genuine tension here: too much KL and you barely improve; too little and you drift into hacked, high-reward-low-quality text. In practice the KL is also monitored during training as an early-warning gauge — a KL that shoots up is the canonical sign the policy has found an exploit. Hold on to this leash idea; it is the thing DPO folds directly into its loss.
PPO in one section
How do you actually maximize an objective whose ‘loss’ comes from sampling text and scoring it with a reward model? You cannot backpropagate through the sampling of discrete tokens, so RLHF reaches for policy-gradient RL, specifically Proximal Policy Optimization (PPO). The loop: sample responses from the current policy, score each with the reward model, fold in the per-token KL penalty to form a shaped reward, estimate advantages (how much better a token was than a learned value baseline expected), and take a gradient step that raises the probability of above-average tokens.
PPO’s signature is the clipped surrogate objective. Let ratio = π_θ(a|s) / π_old(a|s) be the probability ratio between the updated and sampling policies. PPO maximizes min( ratio · A, clip(ratio, 1−ε, 1+ε) · A ), which refuses to reward moving the ratio far outside [1−ε, 1+ε]. That clipping keeps each update small and stable. It works, but it is heavy machinery: you must hold four models in play — policy, reference, reward, and value — run online generation every step, and babysit a famously finicky training loop. That operational weight is precisely what motivates DPO.
The closed-form optimal policy
Here is the pivot on which everything turns. The KL-constrained objective is not just tractable in principle — it has an exact closed-form solution. For any fixed reward r, the policy that maximizes expected reward minus β-scaled KL from π_ref is:
π*(y | x) = (1 / Z(x)) · π_ref(y | x) · exp( r(x, y) / β )
where Z(x) = Σ_y π_ref(y | x) · exp( r(x, y) / β )Intuitively this is the reference policy reweighted by an exponential tilt toward reward. A response keeps its prior probability under π_ref but is multiplied by exp(r/β): high-reward responses get boosted, low-reward ones suppressed, and β controls how sharp the tilt is (small β = aggressive reweighting). The term Z(x), the partition function, just renormalizes so the probabilities sum to one. This is the Gibbs/Boltzmann distribution in disguise, and it is the single most important formula in the DPO story.
Deriving the optimal policy
The closed form is worth deriving, because the derivation shows it is exact, not an approximation. Start from the objective and rewrite the KL explicitly, folding the reward inside a single expectation:
max_π E_y~π [ r(x,y) − β log( π(y|x) / π_ref(y|x) ) ]
= −β · min_π E_y~π [ log( π(y|x) / π_ref(y|x) ) − (1/β) r(x,y) ]
= −β · min_π E_y~π [ log( π(y|x) / ( π_ref(y|x) exp(r(x,y)/β) ) ) ]Now divide and multiply inside the log by Z(x) defined above. The reference-times-exp term becomes exactly Z(x) · π*(y|x), so the bracket turns into log( π(y|x) / π*(y|x) ) − log Z(x). Since Z(x) does not depend on π, the problem reduces to min_π KL( π(y|x) || π*(y|x) ). A KL divergence is minimized — and equals zero — exactly when the two distributions are equal. Therefore the optimum is π = π*, the Boltzmann form above. No gradient descent, no approximation: the constrained problem is solved in closed form.
The partition function problem
If we have π* in closed form, why not just compute it and sample from it? Because of Z(x). The partition function sums π_ref(y|x) exp(r(x,y)/β) over every possible response y — that is every string the model could ever emit, an astronomically large (effectively infinite) set. There is no way to enumerate it and no way to evaluate the normalizer exactly.
This is the same wall that makes energy-based models hard: you can write the unnormalized density trivially, but normalizing it is intractable. It is the reason classical RLHF does not use the closed form and instead runs PPO — policy gradients cleverly sidestep Z(x) because the gradient of the log-policy does not require the normalizer’s value. DPO’s contribution is to sidestep Z(x) a different, cheaper way: not by avoiding it during optimization, but by arranging the algebra so the intractable term cancels analytically before you ever compute anything. That cancellation is the whole trick, and it comes from reading the closed form backwards.
DPO’s key move: solve for the reward
Every derivation so far treated r as known and solved for the optimal π. DPO reverses the arrow. Take the closed-form relation and solve it for the reward instead. From π*(y|x) = (1/Z(x)) π_ref(y|x) exp(r(x,y)/β), take logs and rearrange:
r(x, y) = β · log( π*(y|x) / π_ref(y|x) ) + β · log Z(x)This says something profound: any reward function can be represented by its own optimal policy. The reward is (up to the prompt-dependent constant β log Z(x)) just β times the log-ratio between the aligned policy and the reference. Instead of training a separate reward network and then finding its optimal policy, we can parameterize the reward implicitly through the policy we actually want. Replace π* with our trainable π_θ and define the implicit reward r̂_θ(x, y) = β log( π_θ(y|x) / π_ref(y|x) ). The policy has become its own reward model. All that remains is to fit it to the preference data — and to watch Z(x) disappear.
Substituting into Bradley-Terry: Z cancels
Recall the Bradley-Terry likelihood depends only on the difference of rewards, r(x, y_w) − r(x, y_l). Plug in the expression for reward we just derived, for both the winner and the loser:
r(x,y_w) − r(x,y_l)
= [ β log(π_θ(y_w|x)/π_ref(y_w|x)) + β log Z(x) ]
− [ β log(π_θ(y_l|x)/π_ref(y_l|x)) + β log Z(x) ]
= β log(π_θ(y_w|x)/π_ref(y_w|x))
− β log(π_θ(y_l|x)/π_ref(y_l|x))Both responses share the same prompt x, so they share the same Z(x) — and the two β log Z(x) terms subtract to zero. The intractable partition function is gone, exactly, with no approximation. This is the payoff foreshadowed back at Bradley-Terry: because preferences are shift-invariant in reward, the one quantity we could never compute never needed computing. What remains is a preference probability written entirely in terms of the policy we are training and a frozen reference — both of which we can evaluate exactly for any given sequence.
The DPO loss
Assemble the pieces. The probability the model assigns to the human’s preference is σ of that reward difference, now expressed in log-ratios. Maximizing the likelihood of the observed preferences means minimizing its negative log — the DPO loss:
L_DPO(θ) = − E_(x, y_w, y_l) ~ D [
log σ( β log( π_θ(y_w|x) / π_ref(y_w|x) )
− β log( π_θ(y_l|x) / π_ref(y_l|x) ) ) ]Stare at it, because this is the entire method. There is no reward model, no sampling, no RL loop — just a supervised loss over the static preference dataset. To evaluate it you run four forward passes per example (the policy and the frozen reference, on y_w and y_l), read off the sequence log-probabilities, form two log-ratios, take their β-scaled difference, push it through log-sigmoid, and backpropagate. It trains like ordinary fine-tuning — stable, no online generation, one model to update — yet it is provably optimizing the same KL-constrained reward objective PPO targets. That equivalence, not mere convenience, is why DPO landed so hard.
The gradient and its intuition
The loss is elegant, but its gradient is where the intuition lives. Differentiating and writing the implicit reward r̂_θ(x,y) = β log(π_θ(y|x)/π_ref(y|x)):
∇_θ L_DPO = −β · E [ σ( r̂_θ(x,y_l) − r̂_θ(x,y_w) ) ·
( ∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x) ) ]Three things happen at once. The gradient raises the log-prob of the winner y_w and lowers that of the loser y_l — the direction is exactly what you would hope. And the whole update is scaled by σ(r̂_θ(x,y_l) − r̂_θ(x,y_w)), the probability the implicit reward currently has the pair ranked backwards. Examples the model already orders correctly (winner’s implicit reward well above the loser’s) contribute almost nothing; examples it gets wrong dominate the batch. This adaptive weighting is not a heuristic bolted on — it falls straight out of the log-sigmoid, and it is what prevents the model from wasting capacity on pairs it has already learned or degenerating when the reference and policy agree.