Versioned datasets

Datasets live in git. Every eval run records dataset commit hash. Old runs stay reproducible.

Advertisement

Deterministic runs

Set model temperature to 0 during evals. Same input → same output → comparable scores.

Advertisement

CI integration

- name: run evals
  run: python eval.py --dataset evals/main.jsonl --output results/
- name: compare
  run: python compare.py results/ baseline/ --fail-on-regression 5%