Why it matters

SFT is instruction tuning core. Understanding shapes RLHF pipeline.

Advertisement

The architecture

Format: chat template.

Mask prompt tokens in loss.

Train on response tokens.

SFT flowChat formatrole + contentMask prompt losstrain on responseCross-entropynext token predFoundation for DPO / RLHF; quality data > quantity; datasets like UltraChat, LMSYS-Chat-1M
SFT.
Advertisement

How it works end to end

Loss masking on prompt.

Format matches inference chat template.

Small LR (1e-5 - 3e-5).

Small epochs.