RLHF Scaling Laws

Why it matters

RLHF scaling shapes alignment budget. Understanding shapes strategy.

The architecture

Reward model scaling: larger better.

Preference data: quality > quantity.

PPO compute: modest improvements.

How it works end to end

Gao et al. 2023 scaling laws for reward model over-optimization.

Preference data quality dominant.

PPO diminishing returns.