Why it matters

Reward model quality shapes alignment. Bad reward = bad policy.

Advertisement

The architecture

Preference pairs → BT loss.

Scalar output.

Used by PPO or discarded in DPO.

Reward model mathPreference pairs(a, b, chosen)Bradley-Terry losslogistic modelScalar rewardused by PPOBradley-Terry (1952) preference model; foundation of RLHF reward model
Reward model math.
Advertisement

How it works end to end

Bradley-Terry 1952.

Neural scoring.

Preference pairs.