PPO Fine-Tuning for LLMs

Why it matters

PPO is RLHF historical baseline. Understanding shapes evolution.

The architecture

Reward model separate.

PPO: clipped advantage + KL penalty.

Value model reduces variance.

How it works end to end

Requires reward model + value model.

KL to SFT reference.

Sensitive to hyperparameters.