Evaluating AI systems: the instrument everything rests on
An evaluation set is a collection of inputs with known-good outputs that can be run on demand. It is the only instrument that tells you whether a change helped, and without one every improvement is an opinion and every regression is invisible until a user finds it. Teams understand this and still get it wrong in two specific ways that are worth naming because they are so consistent. The first is drift: the set is built early from what somebody imagined users would send, real usage moves away from it, nobody updates it, and the numbers keep improving while the product gets worse. The second is averaging: a change raises the overall score by two points and destroys performance on one category, which happens to be the category the business cares about, and the aggregate hides it by construction. Both are avoidable and neither is a technical problem.
Where the cases should come from
From work a human already did correctly. Every organisation has an archive of this and almost none of them think of it as a dataset: adjudicated claims, coded records, corrected catalogue entries, resolved exceptions, prior matters and their outcomes.
Building from that archive has two advantages over inventing cases. The outputs are ones the organisation already stands behind, so nobody can argue about whether they are correct. And the distribution is real, including the awkward minority that synthetic cases never contain and that generates most of the eventual complaints.
There is a third benefit that frequently exceeds the first two. Assembling the set surfaces internal disagreement: cases where two people decided differently on similar facts become visible immediately, and that finding is often more valuable to the business than the system being built.
Fifty real cases beat a thousand invented ones
The instinct is to make the set large. Size is the least important property it has. What matters is that the cases are real, that the labels are right, and that the categories you care about are each represented enough to notice a change.
Fifty is enough to start being useful and it is small enough that a domain expert will actually label it. A thousand synthetic cases cost more to produce, measure a distribution nobody sends, and give a misleading impression of rigour.
Growth should come from production. Sampling real inputs on a schedule, labelling them, and adding them to the set is the mechanism that keeps it honest. An hour a week, permanently assigned to someone, is the entire discipline, and its absence is the drift described above.
Slice it by what matters, not by what is easy
Report per category, always. The categories should be the ones the business recognises: document type, customer segment, language, complexity, whatever the operational distinction actually is.
This is what makes a regression visible. A change that improves the common case and degrades the rare one is invisible in an average and obvious in a slice, and the rare case is frequently the expensive one because it is where the money or the risk concentrates.
It also changes the conversation with the business. A single accuracy number invites an argument about whether it is good enough, which nobody can settle. A table showing quality per category invites a decision about which categories are ready to go live, which is answerable and is usually the real question.
Model-graded evaluation, and its limits
Using a model to score outputs is fast, cheap and makes large-scale evaluation possible. It also correlates imperfectly with human judgement, and how imperfectly depends on the task in ways that are not predictable in advance.
The arrangement that works is a hybrid. Model grading for volume, anchored by a subset that humans have labelled, with the correlation between the two checked periodically. When they diverge, the human labels are right and the automated score has started measuring something else.
The failure to avoid is letting the anchor lapse. A team that stops checking correlation is optimising against a grader rather than against quality, and the drift is silent because every number involved keeps looking healthy.
Evaluating an agent is a different exercise
For a system that returns text, the case is an input and a known-good output. For a system that takes actions, the output is a sequence of decisions, and two runs on the same input can reach the same correct answer by different routes with different costs and different risks taken along the way.
So the set has to score more than the destination. Whether the agent stayed inside its permissions, how many steps it took, and whether it took an irreversible action it did not need to are all part of whether the run was good. A scoring scheme that only checks the final answer will pass an agent that reached the right conclusion by deleting something first.
Run it on a schedule, not on demand
An evaluation set that runs when somebody remembers catches nothing. Quality decays for reasons outside the team, notably a provider updating a model or an upstream source changing format, and none of those events produce an error.
A nightly run against fixed cases, with the score recorded over time, converts silent decay into a visible line. It is the single most valuable piece of operational instrumentation for any system of this kind, and it is cheap because the set is small.
It is also the deliverable that outlives the project. When you change model provider in two years, the evaluation set is what tells you whether the change was an improvement. The system will be rewritten; the set, if it was built from real cases, will still be measuring the same thing.
