Why it matters

ORPO simplifies training. One stage vs SFT + DPO. Understanding shapes pipeline.

Advertisement

The architecture

Loss: SFT loss + odds ratio term.

No reference model.

ORPO structureSFT losson chosenOdds ratiochosen vs rejectedCombinedsingle objectiveSkips reference model; single-stage training vs SFT + DPO
ORPO components.
Advertisement

How it works end to end

Odds ratio: log-sigmoid of chosen odds minus rejected odds.

Combines with negative log-likelihood on chosen.

Single stage.