Why it matters

Reasoning evals track frontier. Understanding shapes eval choice.

Advertisement

The architecture

GPQA: graduate-level science.

MMLU-Pro: harder MMLU.

ARC-AGI: novel reasoning.

Reasoning benchmark tiersGPQA DiamondPhD-level scienceMMLU-Proharder MMLUARC-AGInovel reasoningARC-AGI-2 recent; frontier reasoning models (o3, DeepSeek R1) push these
Reasoning evals.
Advertisement

How it works end to end

GPQA Diamond: 448 hard problems.

MMLU-Pro: 12k Qs.

ARC-AGI + ARC-AGI-2.

BBH (BIG-Bench Hard).