Why it matters

Wrong metric implementation gives wrong results. Standard metrics prevent this. Comparable results across teams enable progress.

Advertisement

The architecture

Load: evaluate.load('metric_name'). Returns metric object.

Compute: metric.compute(predictions=..., references=...).

Evaluate libraryLoad metricstandard implementationComputepredictions vs referencesResultreproducibleSame metric name → same computation across runs; cite metric versions in papers
Metric flow.
Advertisement

How it works end to end

Categories: classification (accuracy, F1, precision, recall), generation (BLEU, ROUGE, METEOR, BERTScore), regression (MSE, MAE), IR (MRR, NDCG).

Combining: combine multiple metrics; get all with one call.

Custom: build custom metric via inherit and override.

LLM eval: growing set of LLM-focused metrics (perplexity, HELM benchmarks).