Stage 1: SFT
Instruction-follow via supervised. 10k-100k demonstrations. Warm-start for policy.
Advertisement
Stage 2: Reward Model
Train model to predict preference scores from (prompt, response) pairs. Bradley-Terry loss on ranked pairs.
Advertisement
Stage 3: PPO
Policy π maximizes reward - β·KL(π || π_ref). KL penalty prevents drift. Actor-critic training.