Why architecture matters here

RL fails on reward hacking, over-optimization, and eval blindness. Architecture matters because RM quality + KL constraint + eval govern outcomes.

Advertisement

The architecture: every piece explained

The top strip is the loop. SFT base. Reward model. Rollout. PPO update.

The middle row is safeguards. KL constraint. DPO alternative. Reward hacking. Eval + safety.

The lower rows are ops. Compute. Metrics. Ops — governance + rollback + audit.

RL for LLM — PPO + DPO + reward model + rollout + KL constraintpost-training that shapes behaviorSFT basestarting pointReward modelpreference-trainedRolloutgenerate responsesPPO updatepolicy gradientKL constraintstay close to SFTDPO alternativeno explicit RMReward hackinggaming the RMEval + safetyregression checksCompute3 model copies for PPOMetricsreward + KL + taskOps — governance + rollback + auditregularizesimplifyguardmeasurebudgetwatchwatchoperateoperate
RL for LLM: PPO/DPO pipeline with KL constraint.
Advertisement

End-to-end flow

End-to-end: SFT model rolled out to produce candidate responses. Reward model scores them. PPO updates policy with KL constraint to SFT. Eval confirms lift. DPO alternative skips RM training via preference pairs directly.