Data format
Triples: (prompt, chosen_response, rejected_response). Human or LLM-judged pairs.
Advertisement
JSONL example
{"prompt": "Write a joke", "chosen": "Why did the... good punchline", "rejected": "Weak joke"}Advertisement
Algorithm
Contrastive loss: increase log-prob of chosen, decrease of rejected. Analytical connection to KL-constrained reward maximization.