An AI evaluation framework is the software that turns "is this model good enough?" into a repeatable measurement: it loads a dataset, formats prompts, calls a model, scores the outputs, aggregates the scores and logs everything so two runs can be compared. Public leaderboards are built on such frameworks, and so is every serious release gate for an LLM product.
This article explains what every framework contains, how the main open-source options differ, how scoring methods work, and the statistics needed to tell a real improvement from noise. It assumes nothing beyond basic Python. The architecture of a full evaluation pipeline, from dataset to production feedback, is covered in LLM evaluation architecture; this page focuses on the framework layer and on reading its numbers correctly.
The anatomy every framework shares
Strip any framework down and you find the same six parts.
- Dataset. Items with an input and, usually, a target: a reference answer, a set of acceptable answers, unit tests, or a rubric. It must be versioned, because a changed item changes the score.
- Solver or template. How an item becomes a model call: the prompt template, the number of few-shot examples, the system prompt, tool definitions, and for agents a multi-step loop.
- Model adapter. One interface over hosted APIs and local inference, with batching, retries and rate limits. Decoding settings live here and change results.
- Scorer. Turns an output into a number: exact match after normalisation, the log-likelihood of each choice, running generated code against tests, or a judge model with a rubric.
- Aggregator. Means, confidence intervals, per-slice breakdowns and comparisons with a baseline.
- Log store. Every prompt, output, score and configuration, so you can read failures and rerun exactly the same evaluation later.
Agent evaluations add a sandbox, typically a container, where the model's tool calls and code run safely, and a scorer that inspects the final state of that environment rather than the text.
The anatomy every framework shares
Strip any framework down and you find the same six parts.
- Dataset. Items with an input and, usually, a target: a reference answer, a set of acceptable answers, unit tests, or a rubric. It must be versioned, because a changed item changes the score.
- Solver or template. How an item becomes a model call: the prompt template, the number of few-shot examples, the system prompt, tool definitions, and for agents a multi-step loop.
- Model adapter. One interface over hosted APIs and local inference, with batching, retries and rate limits. Decoding settings live here and change results.
- Scorer. Turns an output into a number: exact match after normalisation, the log-likelihood of each choice, running generated code against tests, or a judge model with a rubric.
- Aggregator. Means, confidence intervals, per-slice breakdowns and comparisons with a baseline.
- Log store. Every prompt, output, score and configuration, so you can read failures and rerun exactly the same evaluation later.
Agent evaluations add a sandbox, typically a container, where the model's tool calls and code run safely, and a scorer that inspects the final state of that environment rather than the text.
The framework landscape
| Framework | Primary job | Good fit |
|---|---|---|
| lm-evaluation-harness (EleutherAI) | Run many academic benchmarks on base and instruction models with log-likelihood and generation tasks defined in YAML | Comparing checkpoints during training or fine-tuning |
| HELM (Stanford CRFM) | Holistic evaluation: many scenarios, several metrics each, standardised prompting | Broad model comparisons beyond accuracy |
| Inspect (UK AI Security Institute) | Tasks built from dataset, solver and scorer; agents, tools and sandboxes; a log viewer | Custom and agentic evaluations, safety evaluations |
| OpenAI Evals | A framework and registry of evals | Teams already standardised on it |
| promptfoo | Config-driven comparisons of prompts and models with assertions, run from a CLI or CI | Prompt regression tests in application repos |
| DeepEval | Unit-test style LLM tests with built-in metrics | Python teams who want evals in their test suite |
| Ragas | Metrics for retrieval-augmented generation such as faithfulness and context recall | RAG pipelines; see RAG evaluation metrics |
Two of these, run from the command line, show the difference in shape. The harness evaluates a model on named benchmarks; Inspect runs a task you wrote:
# lm-evaluation-harness: a Hugging Face model on two benchmarks
lm_eval --model hf --model_args pretrained=EleutherAI/pythia-160m \
--tasks lambada_openai,hellaswag --device cuda:0 --batch_size auto
# Inspect: a task file you define, against a hosted model
inspect eval triage_task.py --model openai/gpt-4ofrom inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import generate
@task
def triage_task():
return Task(
dataset=json_dataset("triage_v3.jsonl"), # records with "input" and "target" fields
solver=generate(),
scorer=model_graded_qa(),
)Choose by the box that is hardest for you. If it is running fifty standard benchmarks on a checkpoint, use the harness. If it is agents with tools in a sandbox, use Inspect. If it is keeping prompts from regressing in an application repository, a config-driven tool in CI is enough. Many teams use two.
Scoring methods and judge validation
Scoring decides what the number means, so choose it before the dataset is finished.
- Exact or normalised match for short answers: lower-case, strip punctuation and articles, then compare. Cheap and deterministic, but brittle to formatting; log the extracted answer so you can see parser failures.
- Log-likelihood multiple choice: score each option's probability under the model and pick the highest. No parsing, but needs token probabilities, so it works on local models and not on most chat APIs.
- Execution-based: run generated code against hidden tests. The standard summary is pass@k, estimated without bias from n samples with c correct.
- Model-graded: a judge model applies a rubric. Flexible and scalable, and only as good as its agreement with humans, which you must measure.
- Pairwise preference: a judge or human picks the better of two outputs; aggregate into win rates. Good for open-ended quality, but sensitive to position and length.
import numpy as np
def pass_at_k(n: int, c: int, k: int) -> float:
"""Unbiased pass@k from n samples, c of which pass (Chen et al., 2021)."""
if n - c < k:
return 1.0
return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
# 20 samples per problem, 5 pass: pass@1 = 0.25, pass@5 ~ 0.81Validate a judge before trusting it. Hand-label 100 to 200 items, run the judge, and compute agreement and Cohen's kappa against the human labels. For pairwise judging, run every comparison in both orders and count only consistent verdicts. Check for a preference for longer answers and for outputs from the judge's own model family. Judge calibration and bias covers these tests in depth.
Statistics: telling signal from noise
An evaluation score is an estimate from a sample, and most reported gaps between models are smaller than their uncertainty. For a binary score with accuracy p on n items, the standard error is sqrt(p(1 - p) / n). At p = 0.8 and n = 500 that is 0.018, so the 95% interval is about plus or minus 3.5 points. A 2-point gain on 500 items is not evidence of anything on its own.
Comparisons are much tighter when they are paired: both models answer the same items, and you analyse per-item differences. Items that both get right or both get wrong cancel out, and only the disagreements carry signal. A paired bootstrap gives an interval for the difference without distributional assumptions:
import numpy as np
def paired_bootstrap(a: np.ndarray, b: np.ndarray, iters: int = 10_000, seed: int = 0):
"""a, b: per-item scores (0/1 or graded) for the same items in the same order."""
rng = np.random.default_rng(seed)
d = b - a
idx = rng.integers(0, len(d), size=(iters, len(d)))
boots = d[idx].mean(axis=1)
lo, hi = np.percentile(boots, [2.5, 97.5])
return d.mean(), lo, hiTwo further corrections matter. If items come in groups, such as several questions about one document, resample whole groups, not items, or the interval is too narrow. If outputs are sampled at non-zero temperature, run each item several times and average per item first; a single sample adds noise on top of sampling error.
Size the dataset before building it. For a paired comparison with 80% power at the 5% level, you need roughly n = 7.84 * var(d) / delta^2 items, where delta is the smallest difference worth detecting and var(d) is close to the share of items on which the two models disagree. If the models disagree on 12% of items and you care about 3 points, that is 7.84 * 0.12 / 0.0009, about 1,050 items. If that is too many, say so in the report: with 400 items, only differences of about 5 points or more are reliably detectable.
Reproducibility and contamination
The same model on the same benchmark can score differently in two frameworks, and both can be correct about what they ran. The differences come from settings that are easy to overlook:
- prompt template, chat template and system prompt;
- number and choice of few-shot examples;
- decoding: temperature, maximum tokens, stop sequences;
- answer extraction: the regular expression that finds "the answer" in a long response;
- scoring: log-likelihood versus generated answer, normalisation rules;
- framework and dataset versions, and the model version behind an API alias.
Record all of them with every run, compare only runs whose configuration matches except for the one thing under test, and never compare your number with a number from a paper or leaderboard unless you reproduce their configuration. Contamination is the other reproducibility problem: public test items can appear in training data, which inflates scores. Prefer private or freshly written items for decisions, and treat a sudden jump on an old public benchmark with suspicion.
Even temperature zero does not make every run identical: hosted APIs and batched inference can return slightly different outputs for the same request. Measure this directly by running the baseline twice and treating the run-to-run difference as a floor below which no change counts as real. If two runs of the same configuration differ by two points on a slice, a two-point change on that slice tells you nothing.
Worked example: choosing a triage model
A support team must choose between its current model A and a cheaper candidate B for ticket triage: assign one of twelve categories and write a one-line summary. They build 400 items from real, anonymised tickets, labelled by two agents who resolved disagreements, and score category by exact match and summary by a judge validated on 150 hand-graded summaries (kappa 0.71 against humans).
In this illustrative run, A gets 324 categories right (81.0%) and B 338 (84.5%). Unpaired, the standard error of the difference is about 2.7 points and the 95% interval, roughly -1.7 to +8.7, includes zero. Paired, the picture sharpens: on 354 items they agree, B alone is right on 30 and A alone on 16. The per-item differences have a mean of 0.035 and a standard error of 0.017, so the interval is about +0.2 to +6.8 points, and McNemar's test on 30 versus 16 gives p of about 0.04.
That is evidence, but thin. Slicing by category shows B's gain is concentrated in billing tickets, and B is worse on the small "security incident" slice, 9 of 14 against A's 12 of 14. Because a missed security ticket is expensive, the team adds 60 more security items before deciding, and makes that slice's accuracy a separate release gate rather than letting it hide in the average.
Failure modes
- Overfitting the eval. Prompts tuned against the same 400 items until the score rises. Keep a held-out split that is only run before release.
- Parser failures counted as model failures. A model that answers "Category: Billing" instead of "billing". Read a sample of failures every time.
- Unvalidated judges. A judge that rewards length or style. Measure agreement with humans and re-measure when the judge model changes.
- Silent API drift. A model alias that now points at a new version. Log the resolved version and pin it where possible.
- Averages that hide slices. A gain overall with a regression on a critical segment. Report slices with their own intervals.
- Too few items. Decisions on gaps smaller than the interval. Size the dataset from the smallest difference you need to detect.
Running evaluations continuously
Run a fast smoke subset on every prompt or code change in CI, the full suite nightly and before release, and a sample of production traffic continuously. Gate releases on a paired comparison against the current baseline with a pre-agreed rule, for example "no slice drops by more than its interval and the overall difference is not negative", rather than on a fixed absolute score. Store logs so a reviewer can open any failing item. For agents, see agent evaluation at scale; for building item sets from your own traffic, see custom LLM benchmarks.
What to do next
- Write down the decision your evaluation must support and the smallest difference that would change it.
- Build 300 to 500 items from real inputs, versioned, with a held-out split.
- Pick the framework for your hardest box: benchmarks, agents or prompt regression.
- Choose the scorer per field; validate any judge against 100 or more human labels.
- Record template, decoding, extraction, framework and model versions with every run.
- Compare models paired, with bootstrap intervals, and report per-slice results.
- Wire a smoke subset into CI and the full suite into the release gate.