What are AI benchmarks? Explained with a school exam

This week Meta's Muse Ember, Anthropic's Claude Fable 5.2 and OpenAI's GPT-6 Cyber all arrived with tables of benchmark scores, so here is what those scores really measure.

In one sentence: An AI benchmark is a fixed set of questions or tasks, with a fixed way of marking the answers, that every AI model can take so their scores can be compared on the same test.

The everyday picture

Think of a school exam that every school in a region uses. Every student gets the same paper. Every paper is marked against the same answer sheet. The results go into one published table.

That table is useful. A parent can compare schools without visiting each one. But a good exam result does not prove a student will be good at a particular job. It shows how they did on that paper, on that day, under those rules.

AI benchmarks work the same way. The models are the students. The benchmark is the exam paper. The scores in a launch announcement are the results table.

How it works, step by step

1. The benchmark (the exam paper) is written once and kept fixed: a set of questions or tasks, such as maths problems, coding jobs or office tasks. One well-known example, MMLU, published in 2020, covers 57 tasks, from elementary maths and US history to computer science and law.
2. The AI model (the student) is given each question and produces an answer. For agent benchmarks, the model may have to click through a website or edit files to finish a task.
3. The scoring rules (the answer sheet and the marker) check each answer. Some tests compare answers to a known correct one. Others run code or check that a file ended up right.
4. The score (the exam result) is usually a percentage of tasks passed, and it goes into a table next to other models' scores.
5. You (the parent reading the table) use the scores to shortlist models, then check the best ones on your own work.

Where the picture stops

• The exam conditions are not always the same. In a school exam, everyone gets the same time and the same rules. In AI, each company often runs the test itself, with its own settings. Meta's evaluation notes for Muse Ember say its model ran at maximum reasoning effort, and that runs of rival models were "best-effort" and may not be tuned for them.
• A model may have seen the paper before. AI models learn from huge amounts of text, and copies of test questions can end up in it. Researchers call this data contamination: the model can score well by remembering, not by working the answer out, which inflates its score.
• Papers get too easy. When most models score near the top, the test stops telling them apart. Stanford HAI reported in 2023 that on the ImageNet image test, the best result improved by only 0.1 percentage points in a year. So researchers keep writing harder papers.

Why it matters for your business

• Faster shortlists. Benchmarks let you narrow dozens of models down to a few without testing them all yourself.
• Lower risk of overpaying. A cheaper model that scores close to an expensive one on the tasks you care about may do the job at a fraction of the cost. Check that before you commit.
• Fewer surprises. High scores do not mean a model is ready for every job. Stanford's 2026 AI Index shows models beating the human baseline on several tests, while agents using a computer, on the OSWorld test, still scored below people.
• Better questions for vendors. Ask who ran the test, with what settings, and whether results on your kind of work are available.

An example

Imagine a 20-person accounting firm choosing an AI model to pull totals and due dates out of supplier invoices. Two models look close in the vendors' results tables, and one is much cheaper.

Instead of trusting the tables alone, the firm writes its own small exam: 50 past invoices, with the correct totals and dates written down by a staff member. Both models sit the same paper, under the same settings. The firm picks the one that gets the most right for the money, and reruns the same 50 invoices whenever it considers switching.

Frequently asked questions

What is a benchmark in AI?

It is a standard test: a fixed set of questions or tasks plus a fixed way of scoring them. Every model can take the same test, so their scores can be compared.

Can I trust the benchmark scores in an AI launch announcement?

Treat them as a starting point. They are often run by the company selling the model, with settings that may differ between models, so check who ran the test and how.

Why do new AI benchmarks keep appearing?

Older tests get too easy as models improve, and some questions leak into training data. Harder, fresher tests are needed to keep telling models apart.

How do I test an AI model for my own business?

Collect a few dozen real examples of the job, with the right answers. Run each model you are considering on the same examples, with the same instructions, and compare how many it gets right.

Related explainers

• What is an LLM?
• Training vs inference
• What are AI hallucinations?
• What is an AI agent?

Sources

• Hendrycks et al., Measuring Massive Multitask Language Understanding (arXiv, 2020)
• Deng et al., Investigating Data Contamination in Modern Benchmarks for Large Language Models (arXiv)
• Stanford HAI, 2026 AI Index: Technical Performance and AI Benchmarks Hit Saturation (2023)
• Meta, Muse Ember evaluation methodology (November 2026)