Why it matters

PPO powered ChatGPT alignment. Complex; DPO offers simpler alternative.

Advertisement

The architecture

Clipped ratio objective.

Advantage estimate.

Per-token log-prob.

PPO for LLMsClipped ratioobjectiveAdvantagefrom rewardPer-tokenlog-prob updateSchulman et al. PPO; complex for LLMs; DPO offers simpler alternative
PPO for LLMs.
Advertisement

How it works end to end

Schulman PPO.

Applied token-level.

DPO alternative.