Reward Model Scaling

Why it matters

Reward model scaling shapes RLHF quality. Understanding shapes design.

The architecture

Reward model size vs quality.

Preference data scale.

Over-optimization curves.

How it works end to end

Gao et al. 2023.

Bradley-Terry loss standard.

Over-optimization: KL to reference regularizes.