Reward Model Training — Preference to Scalar

Why it matters

Reward model quality determines RLHF quality. Understanding shapes RM training.

The architecture

Bradley-Terry loss on preference pairs.

Model outputs reward for chosen > rejected.

How it works end to end

Architecture: LM base + scalar head.

Loss: -log sigmoid(r(chosen) - r(rejected)).

Regularization: prevent reward hacking.