What are AI evals? Explained with a driving test
An AI eval is a set of real test tasks with clear marking rules that checks your AI system is good enough before it goes live, and after every change. Think of it as a driving test.
- Author
Umair Mateen- Published
- Reading time
- 5 min
- Category
- AI
In one sentence: An AI eval is a set of test tasks, taken from the real work an AI system will do, with clear rules for marking each answer, so you can check the system is good enough before you rely on it and check again every time you change it.
The everyday picture
Think of a driving test. Before a learner may drive alone, an examiner takes them on a route with set situations: a busy junction, a pedestrian crossing, parking in a tight space. The examiner marks each one on a sheet. The pass standard is agreed before the test starts, not after.
Nobody gets a licence because they "seem fine" on a quiet road. They pass because they handled the hard parts of a real route, and someone wrote down how they did.
An AI eval is a driving test for your AI system. Your assistant or chatbot is the learner. Your own questions and tasks are the route. A marking method is the examiner. You decide the pass mark first, then you let the results decide whether it goes on the road.
Your AI system works through test cases taken from your real work. A marker scores each answer, and the score against your agreed pass mark decides whether it goes live or needs more work.
How it works, step by step
- Set the pass standard (agree what a safe driver looks like). Write down what "good" means in a way you can check. Anthropic's guidance says success criteria should be specific and measurable: "accurate sentiment classification", not "good performance".
- Build the test set (plan the route). Collect real examples of the work: customer questions, documents to summarise, forms to fill. Add the awkward cases too, such as unclear, very long or off-topic requests, the way a good route includes a tricky junction.
- Run the AI system (the learner drives). The system answers every test case, using the same model, instructions and data it will use for real.
- Mark the answers (the examiner fills in the sheet). Simple checks can be done by code, such as "did it pick the right category?". Judgement calls go to a person or to a second AI model acting as a judge, working from a written marking guide.
- Decide and repeat (pass, or more lessons). If the score meets the pass mark, it goes live. If not, you fix the instructions, data or model and test again. Every later change gets a retest too.
Try it · Put it in order
Put the steps of an AI eval in order
Tap the steps in the order they happen.
Where the picture stops
- A licence is for life; an eval pass is not. A new model version, a changed instruction or new company data can change how the system behaves. OpenAI's guidance says to run evals on every change, not once.
- A learner drives the route once. An AI system should take it many times. The same question can get a different answer on a different run, so one good drive proves little. Anthropic's guidance favours many automatically marked questions over a few marked by hand.
- The examiner may be another AI. AI judges are fast and cheap, but they have habits, such as favouring longer answers. OpenAI advises checking an AI judge against human marks before trusting it at scale.
Why it matters for your business
- Fewer surprises for customers. You find the wrong answers on your test route, not in a customer's inbox.
- Safer upgrades. When a vendor releases a new model, you can rerun the same test and see whether it is better at your work, not just at public benchmarks.
- Decisions you can explain. A written pass mark and a scored test give managers, auditors and clients something concrete. The US standards body NIST's AI Risk Management Framework calls for a test, evaluation, verification and validation (TEVV) approach to AI systems.
- Lower cost of change. Once the test exists, rerunning it is quick, so trying a cheaper model or a new prompt is a measured step rather than a guess.
An example
Imagine a 20-person accounting firm that wants an AI assistant to sort incoming client emails into "invoice", "tax question", "complaint" and "other". Before switching it on, the office manager collects 200 past emails, including some messy ones, and labels each with the right category. The pass mark is agreed up front: the assistant must match the human label on almost all of them, and must never file a complaint as "other". The first run misses several complaints written in a polite tone, so the team adds examples to its instructions and runs the test again. Months later, when their AI provider offers a newer model, they rerun the same 200 emails before switching.
Try it · What would you do?
Your AI provider offers a newer model. What do you do?
Your support assistant has run well for months. Your AI provider releases a newer model that scores higher on public benchmarks, and switching takes one setting.
Pick what you would do. You can change your mind.
Frequently asked questions
What is the difference between an AI eval and a benchmark?
A benchmark is a public test that every model sits, so models can be compared. An eval is your own test, built from your tasks, that checks whether your system does your job well enough.
Do I need to be technical to build an eval?
Not to start one. The most valuable part is a list of real examples with the answer you would accept for each, which the people who do the work today can write. Running it automatically is the technical part.
How many test cases do I need?
There is no fixed number. Start with a set that covers your common requests and your hardest ones, and add every real mistake you find so it is tested next time.
What is "LLM-as-a-judge"?
It means using one AI model to mark another model's answers against a written guide. It is quicker than human marking, but it should be checked against human judgement first.
Related explainers
The DevDrift builds AI automation that is tested on your own work before and after every change: see how.
Sources
Could you explain this to a colleague now?
Mark it and posts that mention it will show a tick. Saved in this browser only.