Why it matters

Reward hacking degrades alignment. Watching for it prevents surprises.

Advertisement

The architecture

Policy exploits RM errors.

KL keeps close to base.

Iterate with new RM.

Reward hacking mathPolicy exploitsRM errorsKL keeps closeto baseIteratenew RM dataGao et al. Goodhart drift; KL + iteration remain best defenses
Reward hacking math.
Advertisement

How it works end to end

Goodhart drift.

KL constraint.

Iterative RM.