DPO Loss and Reference-Relative Preference

DPO Loss and Reference-Relative Preference: experiment guide

This mathematical instrument evaluates one preferred/rejected response pair. Policy log-probability gap means log probability of the preferred response minus log probability of the rejected response under the current policy. Reference log-probability gap means the same difference under a fixed reference policy. Both inputs use natural logarithms and may be positive or negative. A gap of two means a probability ratio of exp(2), not a probability of two. These are sequence-level gaps, not independent probabilities that need to add to one. Absolute sequence likelihoods are deliberately absent because this single-pair objective depends only on their differences.

Worked details 1

Beta multiplies the difference between the two gaps to give the margin m. The pair loss is log(1 + exp(−m)), evaluated with a stable softplus expression. Preference probability is sigmoid(m). The reported gradient differentiates loss with respect to the policy gap while keeping the reference and beta fixed; it is −beta times sigmoid(−m). Curvature is the corresponding second derivative. These are scalar derivatives of the displayed objective, not gradients of all language-model weights. The graph sweeps the policy gap from −20 to 20 and marks your selected input. Its vertical scale adapts to the largest displayed loss; compare the numerical values when changing beta rather than comparing heights across rescaled graphs.

Worked details 2

Reset parameters restores beta 1, policy gap 2 and reference gap 0. The margin is 2, loss is approximately 0.126928 nats, preference probability is 0.880797, and the gradient is approximately −0.119203. Set Reference log-probability gap to 2 while retaining the policy gap: margin becomes zero and loss becomes ln(2), approximately 0.693147. Increase the reference gap to 4: margin becomes −2 and loss becomes 2.126928. The current policy can prefer the chosen response in absolute terms while still falling short of the reference-relative preference. This is why a reference term cannot generally be discarded.

Worked details 3

For another exact experiment, set both gaps to zero and vary beta. Loss remains ln(2) and preference probability remains one half, but gradient becomes −beta/2 and curvature becomes beta squared divided by four. At beta zero the entire graph is flat and all gradients vanish, exposing a degenerate learning boundary. Negative beta, blank fields and nonfinite inputs are rejected; the visible controls bound beta to 0–10 and both gaps to −20–20. No timer is needed because every edit recomputes this direct loss measurement immediately. No preference dataset, optimizer, reward-model training, generation quality, KL estimate, or PPO comparison is simulated. The experiment cannot establish that reducing one pair loss improves general behavior.