Why it matters

DPO simplifies alignment. Standard alternative to PPO now.

Advertisement

The architecture

Preference (a, b, y).

Log-ratio implicit reward.

Direct optimize.

DPO mathPreference(chosen, rejected)Log-ratioimplicit rewardDirect optimizepolicyRafailov et al. 2023; simpler than PPO; widely adopted
DPO math.
Advertisement

How it works end to end

Rafailov 2023.

Log-ratio reward.

Widely adopted.