Why it matters

PPO for LLMs has quirks (long sequences, KL, value model). Understanding shapes stability.

Advertisement

The architecture

Rollout: policy generates.

Score: reward model.

Update: PPO clipped loss + KL.

PPO for LLMRolloutgenerate + rewardAdvantageGAE + KLPPO updateclip + KLValue model + KL to reference + clipping essential for stability
PPO flow.
Advertisement

How it works end to end

Clip: prevent big policy moves.

KL: to reference (SFT) model.

Value model: reduces variance.

Micro-batching for memory.