Short answer
AI benchmarks are standardised tests used to compare AI models. Each measures a specific skill: MMLU tests broad knowledge, GPQA Diamond tests expert-level science, AIME tests competition maths, SWE-bench Verified tests fixing real software bugs, Humanity's Last Exam tests frontier knowledge, ARC-AGI tests novel reasoning, and LMArena ranks models by human votes. Benchmarks are useful signals, but scores can be inflated by training-data contamination and quickly "saturate", so always test models on your own tasks.
Why benchmarks matter
When a new model launches, its announcement is full of scores. Benchmarks give a common yardstick to compare large language models from different labs, track progress over time and spot strengths and weaknesses. For a live comparison of models, see our LLM benchmarks page.
The most cited AI benchmarks
| Benchmark | What it measures | Format |
|---|---|---|
| MMLU / MMLU-Pro | Broad knowledge across many school and professional subjects | Multiple choice |
| GPQA Diamond | Graduate-level biology, physics and chemistry ("Google-proof") | 198 hard multiple-choice questions |
| AIME | Competition maths (American Invitational Mathematics Examination) | Exact numeric answers |
| SWE-bench Verified | Fixing real GitHub issues in Python projects | Code patches checked by tests |
| HumanEval | Writing short Python functions | 164 programming problems |
| Humanity's Last Exam (HLE) | Frontier expert knowledge across many fields | About 2,500 very hard questions |
| ARC-AGI | Solving novel visual puzzles from a few examples | Grid puzzles |
| MMMU | Understanding images, charts and diagrams | Multimodal questions |
| LMArena (Chatbot Arena) | Which model people prefer in blind comparisons | Human votes, Elo-style ratings |
Knowledge: MMLU and MMLU-Pro
MMLU (2020) contains about 16,000 multiple-choice questions across 57 subjects, from law to medicine. Top models now score around 90%, so it no longer separates them well. MMLU-Pro adds harder questions with ten answer options.
Expert science: GPQA Diamond
GPQA questions were written by PhD experts and designed so that skilled non-experts with web access still get most wrong. The Diamond subset of 198 questions is the most quoted version and a common test of reasoning models.
Maths: AIME and beyond
AIME problems require multi-step reasoning with exact answers, making them hard to guess. Scores jumped dramatically once labs trained models with reinforcement learning to reason step by step; see What is reinforcement learning?. Harder maths benchmarks such as FrontierMath now test research-level problems.
Coding: SWE-bench Verified and HumanEval
HumanEval (2021) checks whether generated functions pass unit tests; most frontier models now ace it. SWE-bench gives a model a real repository and a real bug report and checks whether its fix passes the project's tests. The Verified subset of 500 human-checked tasks, released in 2024, is the main benchmark for coding agents and vibe coding tools.
Frontier knowledge: Humanity's Last Exam
Released in January 2025 by the Center for AI Safety and Scale AI, HLE gathered very difficult questions from experts worldwide, keeping only those that leading models failed at the time. It was designed to stay challenging after older benchmarks saturated.
Novel reasoning: ARC-AGI
Created by François Chollet, ARC-AGI shows a few example grid transformations and asks the model to infer the rule for a new grid. The puzzles are easy for most people but hard for AI because they cannot be memorised. ARC-AGI-2, launched in 2025, raised the difficulty further. See What is AGI? for why this matters.
Human preference: LMArena
LMArena shows users two anonymous answers to the same prompt and asks which is better. Thousands of votes produce a leaderboard. It captures helpfulness and style but can reward length and formatting over correctness.
How to read benchmark scores
- Check the exact setting: pass@1 (one attempt) is stricter than pass@k or "with majority voting".
- Tools allowed? Scores with code execution or web search are not comparable with scores without them.
- Reasoning effort: reasoning models often report results at their highest thinking budget, which is slower and costlier.
- Who ran it? Independent evaluations are more trustworthy than self-reported numbers.
- Small differences: a gap of one or two points is often within noise.
Why benchmarks can mislead
- Contamination: test questions leak into training data from the web, so a model may have memorised answers.
- Saturation: once top models score 90%+, the benchmark stops distinguishing them.
- Goodhart's law: when a score becomes a target, labs optimise for it, sometimes at the expense of real usefulness.
- Narrow coverage: no benchmark captures everything you care about, such as writing quality, honesty, safety or speed.
- Cost and latency ignored: a model that scores slightly higher may be far more expensive to run. See how much energy AI uses.
How to evaluate models for your own use
1. Collect 20 to 100 real examples of your task with good reference answers. 2. Run several candidate models with the same prompt and settings. 3. Score answers for correctness, format and usefulness, by hand or with a strong model as judge, spot-checked by a person. 4. Weigh quality against cost, speed and privacy, including small language models you can run locally.
Hardware benchmarks are different
AI hardware is measured with benchmarks such as MLPerf (training and inference speed on standard models), tokens per second and cost per million tokens. These depend on GPU memory bandwidth, precision and software. See GPU memory bandwidth, the LLM inference simulator and our GPU cloud price tracker.
Frequently asked questions
Which benchmark is most important?
It depends on your use case: SWE-bench for coding, GPQA and HLE for expert knowledge, AIME for maths, MMMU for images, and LMArena for general chat preference.
Why do new benchmarks keep appearing?
Models improve so quickly that older tests saturate within a year or two, so researchers create harder ones to keep measuring progress.
Can companies cheat on benchmarks?
Deliberate cheating is rare, but contamination, selective reporting and tuning for specific benchmarks happen. Independent and held-out evaluations help.
Do higher benchmark scores mean a model is smarter?
They mean it performs better on those tests. Real-world usefulness also depends on reliability, speed, cost, safety and fit for your task.
Sources and further reading
- Hendrycks et al., Measuring Massive Multitask Language Understanding (MMLU, 2020)
- Rein et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023)
- Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023)
- Phan et al., Humanity's Last Exam (2025)
- Chollet, On the Measure of Intelligence (ARC, 2019)
- MLCommons MLPerf benchmarks




Leave a Reply