Skip to content

AI Benchmarks Explained: MMLU, GPQA, SWE-bench, Humanity's Last Exam and More

What do AI benchmark scores like MMLU, GPQA Diamond, SWE-bench Verified, AIME, Humanity's Last Exam, ARC-AGI and LMArena actually measure? A plain-English guide to the most cited LLM benchmarks, how to read them, and why scores can mislead.

5 min read

AI Benchmarks Explained — cover image from TensorCore.si

Short answer

AI benchmarks are standardised tests used to compare AI models. Each measures a specific skill: MMLU tests broad knowledge, GPQA Diamond tests expert-level science, AIME tests competition maths, SWE-bench Verified tests fixing real software bugs, Humanity's Last Exam tests frontier knowledge, ARC-AGI tests novel reasoning, and LMArena ranks models by human votes. Benchmarks are useful signals, but scores can be inflated by training-data contamination and quickly "saturate", so always test models on your own tasks.

Why benchmarks matter

When a new model launches, its announcement is full of scores. Benchmarks give a common yardstick to compare large language models from different labs, track progress over time and spot strengths and weaknesses. For a live comparison of models, see our LLM benchmarks page.

The most cited AI benchmarks

BenchmarkWhat it measuresFormat
MMLU / MMLU-ProBroad knowledge across many school and professional subjectsMultiple choice
GPQA DiamondGraduate-level biology, physics and chemistry ("Google-proof")198 hard multiple-choice questions
AIMECompetition maths (American Invitational Mathematics Examination)Exact numeric answers
SWE-bench VerifiedFixing real GitHub issues in Python projectsCode patches checked by tests
HumanEvalWriting short Python functions164 programming problems
Humanity's Last Exam (HLE)Frontier expert knowledge across many fieldsAbout 2,500 very hard questions
ARC-AGISolving novel visual puzzles from a few examplesGrid puzzles
MMMUUnderstanding images, charts and diagramsMultimodal questions
LMArena (Chatbot Arena)Which model people prefer in blind comparisonsHuman votes, Elo-style ratings

Knowledge: MMLU and MMLU-Pro

MMLU (2020) contains about 16,000 multiple-choice questions across 57 subjects, from law to medicine. Top models now score around 90%, so it no longer separates them well. MMLU-Pro adds harder questions with ten answer options.

Expert science: GPQA Diamond

GPQA questions were written by PhD experts and designed so that skilled non-experts with web access still get most wrong. The Diamond subset of 198 questions is the most quoted version and a common test of reasoning models.

Maths: AIME and beyond

AIME problems require multi-step reasoning with exact answers, making them hard to guess. Scores jumped dramatically once labs trained models with reinforcement learning to reason step by step; see What is reinforcement learning?. Harder maths benchmarks such as FrontierMath now test research-level problems.

Coding: SWE-bench Verified and HumanEval

HumanEval (2021) checks whether generated functions pass unit tests; most frontier models now ace it. SWE-bench gives a model a real repository and a real bug report and checks whether its fix passes the project's tests. The Verified subset of 500 human-checked tasks, released in 2024, is the main benchmark for coding agents and vibe coding tools.

Frontier knowledge: Humanity's Last Exam

Released in January 2025 by the Center for AI Safety and Scale AI, HLE gathered very difficult questions from experts worldwide, keeping only those that leading models failed at the time. It was designed to stay challenging after older benchmarks saturated.

Novel reasoning: ARC-AGI

Created by François Chollet, ARC-AGI shows a few example grid transformations and asks the model to infer the rule for a new grid. The puzzles are easy for most people but hard for AI because they cannot be memorised. ARC-AGI-2, launched in 2025, raised the difficulty further. See What is AGI? for why this matters.

Human preference: LMArena

LMArena shows users two anonymous answers to the same prompt and asks which is better. Thousands of votes produce a leaderboard. It captures helpfulness and style but can reward length and formatting over correctness.

How to read benchmark scores

  • Check the exact setting: pass@1 (one attempt) is stricter than pass@k or "with majority voting".
  • Tools allowed? Scores with code execution or web search are not comparable with scores without them.
  • Reasoning effort: reasoning models often report results at their highest thinking budget, which is slower and costlier.
  • Who ran it? Independent evaluations are more trustworthy than self-reported numbers.
  • Small differences: a gap of one or two points is often within noise.

Why benchmarks can mislead

  • Contamination: test questions leak into training data from the web, so a model may have memorised answers.
  • Saturation: once top models score 90%+, the benchmark stops distinguishing them.
  • Goodhart's law: when a score becomes a target, labs optimise for it, sometimes at the expense of real usefulness.
  • Narrow coverage: no benchmark captures everything you care about, such as writing quality, honesty, safety or speed.
  • Cost and latency ignored: a model that scores slightly higher may be far more expensive to run. See how much energy AI uses.

How to evaluate models for your own use

1. Collect 20 to 100 real examples of your task with good reference answers. 2. Run several candidate models with the same prompt and settings. 3. Score answers for correctness, format and usefulness, by hand or with a strong model as judge, spot-checked by a person. 4. Weigh quality against cost, speed and privacy, including small language models you can run locally.

Hardware benchmarks are different

AI hardware is measured with benchmarks such as MLPerf (training and inference speed on standard models), tokens per second and cost per million tokens. These depend on GPU memory bandwidth, precision and software. See GPU memory bandwidth, the LLM inference simulator and our GPU cloud price tracker.

Frequently asked questions

Which benchmark is most important?

It depends on your use case: SWE-bench for coding, GPQA and HLE for expert knowledge, AIME for maths, MMMU for images, and LMArena for general chat preference.

Why do new benchmarks keep appearing?

Models improve so quickly that older tests saturate within a year or two, so researchers create harder ones to keep measuring progress.

Can companies cheat on benchmarks?

Deliberate cheating is rare, but contamination, selective reporting and tuning for specific benchmarks happen. Independent and held-out evaluations help.

Do higher benchmark scores mean a model is smarter?

They mean it performs better on those tests. Real-world usefulness also depends on reliability, speed, cost, safety and fit for your task.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *