How do you measure what an AI model can do?

Evaluating a model means running it over a fixed set of tasks with known answers and counting how many it gets right. The number that comes out depends on four things: which tasks, how the prompt was assembled, who grades, and how the results are aggregated. Change any one of them and the score moves without the model having changed.

That fragility is not sloppy execution, it is the nature of the exercise. Measuring the capability of a system that accepts any natural-language input requires picking a minuscule sample of what it can do and hoping the sample represents the rest. How a benchmark is built is the question that decides whether that bet has a chance.

This piece is the map of the subject. The four families of evaluation, the journey of a score from question to table row, and the pitfalls that make benchmarks mislead even when nobody is acting in bad faith. What each specific leaderboard measures lives in MMLU, GPQA and SWE-bench explained.

The four families

The division that matters is not by subject. It is by who decides an answer is correct, because that determines what the number can prove.

Knowledge uses multiple choice against an answer key. MMLU, from 2020, is the example that set the format: questions across 57 subjects, from elementary mathematics to law and ethics, drawn from real exam material and designed to demand world knowledge rather than linguistic skill¹. Grading is string comparison, which makes this family cheap to run and easy to contaminate.

Reasoning also has an answer key, but reaching it takes a chain of steps. GPQA pushed that to the limit in 2023: 448 biology, physics and chemistry questions written by people with or working toward doctorates, where domain experts score 65% and non-expert validators with unrestricted web access score 34% after over half an hour per question. The strongest GPT-4-based baseline of the time reached 39%². The design exists precisely so that searching the web does not solve it.

Agentic swaps the answer key for a verifier. SWE-bench, also from 2023, hands the model a repository and a real issue description and asks for a patch; grading means running the test suite. It holds 2,294 problems drawn from issues and pull requests across 12 popular Python projects, and at launch the best model resolved 1.96% of them³. This family is the most expensive to run and the hardest to fake: tests do not pass by accident.

Preference has no correct answer at all. Two models answer the same question, someone picks the better one, and a rating system turns those picks into a ranking. The Arena published its method in 2024 with over 240 thousand votes collected⁴. It measures what pleases, which is real information and is not the same thing as being right.

The path a score travels

A benchmark number looks like a fact and is the outcome of four chained decisions. All four are worth walking through, because each has a documented failure mode.

The question may already be in the training data

Contamination is the best known problem and the worst solved. The usual defence is stripping from the training corpus anything that overlaps the test, measured by n-gram overlap. A 2023 paper showed that is not enough: simple variations of the test, such as paraphrase or translation, walk through those filters, and a 13-billion-parameter model trained on them reaches performance on par with GPT-4 on the contaminated benchmark. The authors applied stronger detection to known pretraining sets and found 8% to 18% of HumanEval overlapping in RedPajama-Data-1T and StarCoder-Data⁵.

The most elegant experiment on the size of the effect came in 2024. A group commissioned a thousand fresh problems in the style and difficulty of GSM8K, the grade-school arithmetic benchmark, matching human solve rate, number of solution steps and answer magnitude. Models across several families dropped by up to 8 points when the known test was swapped for the new one, and the drop correlated with the model’s probability of generating GSM8K examples from memory. The honest part of the result: many models, especially frontier ones, showed little sign of overfitting⁶. Contamination is real and does not explain everything.

The answer moves with how you ask

This is the factor that most damages public comparison and least appears in promotional material. The same model, on the same benchmark, produces different scores depending on how the prompt is assembled, how many examples ride along, whether the answer is chosen by comparing probabilities across options or generated as free text, and which rule extracts the answer from that text.

The team maintaining one of the most used harnesses published a summary of three years of exactly this in 2024: model sensitivity to the evaluation setup, difficulty comparing across methods, and a lack of reproducibility and transparency, with concrete instances where missing best practice distorted results⁷. The practical recommendation that falls out of it is dull and effective: publish the configuration alongside the number, or do not publish the number.

The eval harness is the piece of software making those decisions, and it never shows up in the table.

The grading may be wrong

Answer keys are written by people, and people make mistakes. A 2024 study manually re-annotated 5,700 questions across all 57 MMLU subjects and estimated that 6.49% of the benchmark contains answer-key errors. The concentration is the alarming part: in the virology subset, 57% of the analysed questions had a problem. Recomputing scores on the corrected version produced significant discrepancies against what had originally been reported⁸.

That carries an uncomfortable implication. When a model sits a few points below a benchmark’s ceiling, part of the remaining gap consists of questions whose “correct” answer is wrong. Optimising against that remainder rewards reproducing the annotator’s mistake.

The number ships without a margin

The last stage is aggregation, and it is where the most useful information gets lost. A score of 87.3% over 448 questions carries a sampling uncertainty that is rarely published. A 2024 article treated evaluations as what they are, experiments, and showed how to compute the variance of a measurement, how to compare two models accounting for the fact that they answered the same questions, and how to size how many questions are needed to detect a difference of a given size⁹.

The practical consequence fits in one sentence: a one-point difference between two models on a benchmark of a few hundred questions is usually noise, and that is exactly the kind of difference that becomes a headline.

Human preference measures something else

Comparison-based evaluation solves the answer-key problem at the cost of trading “correct” for “preferred”. The Arena documented its method in 2024 and showed that crowd votes agree well with expert raters on the questions collected⁴. A variant automates the judge: a strong model compares the two answers instead of a person, which makes the operation cheap and imports the judge’s biases, such as favouring long answers or whichever came first¹⁰.

In 2025 the structural critique arrived. A study analysed the Arena’s practices and identified undisclosed private testing that benefits some providers, who can evaluate several variants before public release and choose which score to disclose. The authors identified 27 private variants tested by a single provider in the run-up to a release, and estimated that two providers received roughly 19.2% and 20.4% of all platform data, while 83 open-weight models combined got 29.7%¹¹. The effect described is overfitting to Arena dynamics rather than general quality.

None of that makes the ranking useless. It makes it a signal with a known bias, which is something you use carefully rather than something you cite as a verdict.

The map of the subject

Build decides where the questions come from, who writes the answer key and how the test is kept out of training. It is the stage that sets the quality ceiling for everything downstream.

Run is the harness, the prompt format and how many samples per question. Cheap to get right, and responsible for most of the discrepancies between published numbers.

Interpret covers error bars, comparison across versions and the special case of long-context benchmarks, where the gap between advertised window and usable window only shows up if the test was designed to find it.

Pitfalls is the category that good engineering does not remove, because it is made of incentives: training on the test, benchmark saturation, and the selection that happens when whoever publishes the number also chooses which number to publish. Preference ranking is where that dynamic is best documented.

Where it breaks

Benchmarks saturate and stop informing. Once the top models clear 90%, what is left to measure is the residue, which is made of ambiguous questions and answer-key errors. From then on the score separates models for the wrong reason.

The average hides the distribution. A model at 80% might ace 56 of MMLU’s 57 subjects and score zero on one. For your application, which probably lives inside a single subject, those two 80% models are not interchangeable.

Average capability does not predict your case. This is the most expensive failure in practice. The benchmark measures short tasks, clean inputs and a fixed criterion; your application has dirty inputs, its own context, and a criterion nobody has written down yet.

What is not measured disappears. Latency, cost per call, format stability, behaviour on adversarial input and refusal rate rarely enter the leaderboard, and they decide whether the system ships.

The life cycle of a benchmark

Every benchmark follows the same arc, and recognising it helps you read any leaderboard by its date. It is born hard, with the best models on the floor. It becomes an optimisation target. It climbs fast, partly through real capability and partly through contamination. It saturates above 90%. And it gets retired, replaced by something harder, while continuing to appear in marketing material for another year.

SWE-bench illustrates the first half of the arc: 1.96% at launch in 2023³, and two years later it was the test every lab quoted. MMLU illustrates the second: in 2025 a group launched a new benchmark arguing explicitly that popular tests were not keeping pace in difficulty, since models were clearing 90% on MMLU, and assembled 2,500 questions at the frontier of human knowledge written by subject-matter experts across dozens of fields to get signal back¹².

The consequence for a reader is direct. A high score on a mature benchmark tells you little, because everybody is high. A score on a freshly launched benchmark tells you more and compares less, because there is no history yet. Neither substitutes for measuring your own case.

How to evaluate your case

1. Write 30 to 50 real cases before looking at any leaderboard. Real input, real expected output. It is the only instrument that answers your question.
2. Define the grading rule before you run. If you cannot write down what counts as correct, the number that comes out will not mean anything.
3. Run each case more than once. Above zero temperature the answer varies, and a single run measures luck alongside capability.
4. Compute the margin before deciding. Across 40 cases, a two-answer gap between two models is not a gap.
5. Measure what public benchmarks ignore. Cost per call, time to first token and malformed-output rate, in the same spreadsheet as accuracy.
6. Freeze the set and the configuration. Changing the cases and the model in the same week makes it impossible to know what moved the number.

What it costs

In orders of magnitude: a set of 50 cases, run three times across three models, is 450 calls. With prompts of a few thousand tokens that costs a handful of dollars and finishes in minutes. The real cost is human, and it sits in writing the expected outputs and reviewing the disagreements: counting a few hours of work from someone who knows the domain is more realistic than counting tokens.

Agentic evaluation changes the arithmetic. Each case can involve dozens of calls, code execution in a sandbox and minutes of wall-clock time, which puts a full run in the tens or hundreds of dollars. That is why this family gets run less often, and why automating execution is worth more than adding cases.

Footnotes

1. Hendrycks et al. (2020) built a multiple-choice test covering 57 subjects, from elementary mathematics to law and ethics, designed to demand world knowledge rather than linguistic skill. ↩

2. Rein et al. (2023) wrote 448 biology, physics and chemistry questions validated by experts: 65% accuracy among people holding or pursuing doctorates in the field, 34% among non-expert validators with unrestricted web access, and 39% for the strongest GPT-4-based baseline of the time. ↩

3. Jimenez et al. (2023) assembled 2,294 problems from real issues and pull requests across 12 Python repositories, graded by running the test suite. At launch the best model resolved 1.96%. ↩ ↩²

4. Chiang et al. (2024) described the pairwise comparison platform with over 240 thousand votes collected, and verified that crowd votes agree with expert raters. ↩ ↩²

5. Yang et al. (2023) showed that paraphrase and translation of the test walk through text-overlap decontamination filters, and found 8% to 18% of HumanEval overlapping in public pretraining sets. ↩

6. Zhang et al. (2024) commissioned a thousand fresh problems in GSM8K’s style and measured drops of up to 8 points, correlated with the model’s probability of generating the original benchmark’s examples from memory. ↩

7. Biderman et al. (2024) documented, from three years maintaining an evaluation harness, model sensitivity to the test setup and the lack of reproducibility across implementations. ↩

8. Gema et al. (2024) manually re-annotated 5,700 questions across all 57 MMLU subjects, estimating 6.49% answer-key error across the benchmark and 57% in the virology subset analysed. ↩

9. Miller (2024) treated evaluations as statistical experiments and derived formulas for variance, comparison between two models over the same questions, and sizing the number of questions. ↩

10. Zheng et al. (2023) measured agreement between automatic judges and human preference, and catalogued the judge’s biases, including a preference for long answers and for position. ↩

11. Singh et al. (2025) identified undisclosed private testing on the Arena, including 27 variants tested by a single provider ahead of a release, and asymmetric data access between closed providers and open-weight models. ↩

12. Phan et al. (2025) assembled 2,500 questions written by subject-matter experts across dozens of fields, arguing that popular benchmarks stopped informing once models cleared 90% on tests like MMLU. ↩