Eval

Definition: what an eval is and is not

An eval is a fixed set of inputs, a defined way to run them, and a scoring method that produces comparable results over time. For an AI agent, the input is a scenario: a caller who wants to reschedule a delivery, a WhatsApp user asking about a refund, an email reply that contains a complaint. The output is a trace, and the score is attached to what the agent did, not only to what it said.

It is not a demo. A demo is a curated path through the happy case, performed once, in front of someone who wants to be convinced. An eval runs the unhappy cases repeatedly, without a human choosing the next prompt, and reports the rate at which the system succeeds. The difference is not rigour for its own sake; it is that production traffic is mostly unhappy cases, and those are the ones nobody rehearsed.

It is also not a public benchmark. A leaderboard measures a model against someone else’s task. An eval measures your system, with your tools, your policy, your languages and your customers, against your definition of correct. A model that tops a benchmark can still fail your refund flow because it does not know your order system’s error codes.

Eval versus unit test: the difference that changes design

A unit test is deterministic. Given this input, the function returns this value, and a mismatch is a bug. An eval is probabilistic. The same input can produce different behaviour across runs, model versions and sampling settings, so the result is a rate with uncertainty, not a boolean. Treating an eval like a unit test produces two failure modes: flaky pipelines that block releases randomly, and false confidence from a single green run.

The design consequences are practical. Unit tests run on every commit because they are fast and cheap. Evals run on a schedule, on a pull request that touches prompts or tools, and before a release, with a sample size chosen so that the difference you care about is detectable. If your suite has twenty cases, a change from 85 to 90 percent is noise. If it has four hundred, the same movement is a signal worth acting on.

Both are needed, and they answer different questions. Unit tests protect the code path: does the tool call use the right schema, does the guardrail block the forbidden action, does the handoff produce a valid payload. Evals protect the behaviour: does the agent actually resolve the request, does it escalate when it should, does it capture consent before sending a template. A team with only unit tests ships a working machine that behaves badly.

Golden sets: provenance is the whole game

A golden set is a curated collection of scenarios with expected outcomes. The word golden implies a standard, and the standard is only as good as where the cases came from. The best sets are sampled from real traffic, redacted, labelled by someone who knows the correct outcome, and versioned so a score can be tied to the exact set that produced it. A set written by the same person who wrote the prompt tends to test the prompt’s assumptions rather than the customer’s reality.

Composition matters as much as size. A voice agent for Indian customers needs Hinglish code-switching, real names, spelled-out numbers, background noise and interruptions. A WhatsApp agent needs template boundaries, media messages and the point where the customer service window closes. An email agent needs long threads, forwarded chains and attachments that change the meaning of the request. If the set is all clean English text, the score is a statement about clean English text.

Then there is maintenance. Behaviour drifts as models change, as tools change and as the business changes its policy. Cases that were correct in March may be wrong in October because the refund rule changed. Review the set on a schedule, retire stale cases with a note, and keep a holdout portion that nobody tunes against. A golden set that has been edited after every failing run is no longer a measurement; it is a mirror.

• Sample from production traces, then redact and label.
• Stratify by channel, language, intent and customer segment.
• Keep a holdout slice that is never used for tuning.
• Version the set and record the version with every score.
• Retire cases when policy changes, with a written reason.

Rubric scoring, and why checklists beat vibes

A rubric converts a judgement into a small number of observable criteria. For a support agent, that might be: identified the order, stated the correct policy, offered the available options, captured consent where required, did not promise a refund it cannot authorise, and closed with a clear next step. Each criterion is scored independently, so a failure points at a behaviour rather than at a general impression of quality.

Two kinds of rubric are useful. Binary checks are unambiguous: did the agent call the order lookup tool before quoting a delivery date. Graded rubrics handle qualities that exist on a spectrum, such as clarity or empathy, usually on a three or five point scale with anchored descriptions for each level. Binary checks are more reliable and should carry most of the weight; graded scores are useful for tracking tone drift but are easy to argue about.

The rubric needs a decision rule before it is used. What counts as a pass on a six-criterion case: all six, or five with no critical failure. Critical criteria, such as consent capture or refusal to give medical advice, should be able to fail a case on their own. Writing that rule down prevents the familiar meeting where two reviewers look at the same transcript and disagree about whether it was good.

Model judges, and how to keep them honest

Human labelling does not scale to thousands of runs, so teams use a model as a judge. The judge receives the scenario, the trace and the rubric, and returns scores with reasons. It is fast and cheap, and it is also biased in ways that are well documented: it tends to prefer longer answers, it can favour the first option in a pairwise comparison, and it may prefer text that resembles its own output.

The mitigation is calibration, not faith. Label a few hundred cases by hand, run the judge on the same cases, and measure agreement per criterion. Where agreement is poor, tighten the rubric, add worked examples of each score, or replace the judge with a deterministic check. Report judge agreement alongside the eval score, because a score produced by an uncalibrated judge is a number with no error bar.

Use judges where the criterion is genuinely qualitative and keep deterministic checks where they are possible. Whether the agent called the lookup tool, whether the JSON matched the schema, whether the consent flag was set, whether the response stayed under a length limit: these are code, not judgement. A healthy eval suite is mostly deterministic, with a calibrated judge for the parts a test cannot see.

What an agent eval has to cover

Agent evals fail when they only score the text. An agent’s behaviour includes the plan it formed, the tools it called, the arguments it passed, the state it changed and the handoff it produced. A polished reply that quotes the wrong order is a failure. A correct reply that bypasses the human gate on a high-value refund is a worse failure, because it is a policy breach that reads well.

The core criteria are consistent across channels. Task success: did the customer’s request get resolved, and was the resolution correct. Tool correctness: right tool, right arguments, no unnecessary calls. Policy compliance: did the agent stay inside the rules it was given, including refusals. Escalation: did it hand off at the right moment, with the right context, rather than looping or giving up. Consent and identity: did it verify who it was talking to and capture permission before acting.

Then add the operational criteria that decide whether the thing is usable: latency at p95, cost per resolved task, and containment, meaning the share of conversations finished without a human. A model that resolves everything at four times the latency is a product decision, not a win. Score these alongside quality so the trade-off is visible in one report rather than discovered in production.

Regression gates that do not cry wolf

A regression gate blocks a change when a score falls below a threshold. The design question is which threshold, on which slice, with which sample. A single overall threshold hides the failures that matter: an agent can improve its average while getting worse at escalation, because escalation is a small share of cases. Gate on slices, and give critical slices their own, stricter limits.

Statistical honesty is the second requirement. With small samples, scores move for no reason. Estimate the noise floor by running the same suite twice against an unchanged system, then set thresholds outside that band. If the suite cannot distinguish a real regression from sampling noise, the answer is more cases in the slice, not a tighter threshold that fails randomly.

The third is cost and speed. A full suite on every commit is usually too slow and too expensive, so split it: a fast smoke set of high-risk scenarios on every pull request, the full suite nightly and before a release, and a deep evaluation of a new model version before it is promoted. Every failure needs an owner and a triage path, because a gate nobody respects is worse than no gate at all.

Online evaluation and the drift problem

Offline evals answer questions about the cases you thought of. Production answers questions about the cases you did not. Online evaluation samples real conversations, scores them with the same rubric, and watches for changes in the mix: a new campaign brings a new intent, a new city brings a new language pattern, a policy change makes yesterday’s correct answer wrong.

Sample deliberately. Score every conversation that involved a tool failure, a human handoff or a negative sentiment signal, and a random slice of the rest. That design catches rare failures without paying to score everything. Keep the sampled traces linked to the model version, prompt version and tool version, so a shift in scores can be attributed to a change rather than to the weather.

Close the loop. Traces that fail should become candidate cases for the golden set after redaction and labelling, which is how a suite stays aligned with reality. The most common failure of eval programmes is not a bad judge; it is a suite that stopped changing while the product did.

Mistakes that make an eval lie

The first mistake is tuning on the test set. It happens innocently: a case fails, someone edits the prompt until it passes, and repeats. After twenty iterations the score is high and the system has learned the set. Keep a holdout, refresh cases regularly, and treat a suspiciously fast improvement as a warning rather than a win.

The second is a rubric that scores style instead of outcomes. An agent that writes beautifully and resolves nothing will pass a tone rubric. Weight task success and policy compliance above fluency, and make critical failures fatal. The third is a judge that was never calibrated, which produces confident numbers that move with the judge’s preferences rather than with the product.

The fourth is coverage theatre: a large suite drawn from one channel, one language and one customer type, reported as a single percentage. Slice the report. A 92 percent overall score that hides 71 percent on Hinglish voice calls is not a score, it is a blind spot with a decimal point. The fifth is no owner. An eval suite without a named owner decays in a quarter, and everyone keeps quoting the last green run.

A worked example, and the numbers to report

Consider a support agent handling 60,000 conversations a month across WhatsApp, email and voice, with a golden set of 600 cases: 200 per channel, stratified by intent, with 120 Hinglish voice cases. The first run scores 89 percent overall. The slice report shows 96 percent on email, 91 on WhatsApp, 88 on English voice and 64 on Hinglish voice. The overall number was never the useful number.

The team fixes the Hinglish slice by adding real audio, better number handling and a confirmation step for names and addresses. Four iterations later the slice reaches 84 percent and the overall score moves by three points, which is within noise. That is the correct outcome: the work improved the experience for a group of customers without inflating a headline.

Report five things every release: overall task success with the sample size, the weakest slice, critical-failure count, judge agreement, and p95 latency with cost per resolved task. That report is short enough to read in a standup and specific enough to argue with. It also makes the trade-offs explicit, which is the only way an eval programme survives contact with a roadmap.