Data format
JSONL: each line is {messages: [{role, content}]}. Chat format standardized. Alpaca simpler (instruction/input/output).
Advertisement
Loss
Cross-entropy on assistant tokens. Mask user + system tokens from loss (typical). Only train on what model should generate.
Advertisement
Batch size + LR
Small learning rate (1e-5 to 5e-5). Small batch on GPU (4-32 with gradient accumulation to effective 128+). Overfits fast.