Before a new hire answers customers alone, someone checks their replies for a while and decides when they're ready. An AI agent needs the same treatment, in a more structured form. That structure is called an eval.

Why agents need a different kind of testing

Ordinary software is predictable. Click the same button with the same data and you get the same result, so a tester can check it once and move on.

AI agents don't work that way. They're built on language models, and the same question can produce a slightly different answer each time. Anthropic's engineering team says it plainly in its guide to evaluating AI agents: "Because model outputs vary between runs, we run multiple trials to produce more consistent results." One good demo tells you very little.

Agents also fail differently. A broken form shows an error. A wrong answer from an agent usually reads just as confidently as a right one.

And those mistakes are yours. In Moffatt v. Air Canada, the airline's chatbot told a customer he could apply for a bereavement fare retroactively, contradicting another page on the airline's own site. Air Canada argued it couldn't be held responsible for its chatbot. B.C.'s Civil Resolution Tribunal disagreed: "It makes no difference whether the information comes from a static page or a chatbot," as quoted in McCarthy Tétrault's summary of the decision. Whatever your agent tells a customer, your business has said.

What an eval is

An eval is a set of real example cases, each paired with a description of what a good answer looks like, plus a consistent way to grade the agent's response. You run the agent through every case, score the results and study the failures. Instead of "seems fine to me," you get a record of what works and what doesn't.

The eval is the spec

Ordinary software starts with a specification: what it should do and what it shouldn't. For an agent, the most useful specification is the eval itself. Each case says, in concrete terms, "when a customer writes this, a good reply does that."

That leads to a simple rule: if you can't write the test cases, you're not ready to automate. The exercise often reveals that two experienced staff handle the same request differently, or that a policy lives only in one person's head. Better to settle that in a meeting than in front of customers.

Writing evals is business work, not technical work. The best people for it are the ones who answer those emails today.

Build it from your own history

You already have the raw material: past emails, support tickets, quotes and chat logs. Real wording beats invented wording. Pull a mix:

  • Everyday questions that make up most of your volume.
  • Awkward edge cases: the angry customer, the request for an exception, the email asking three things at once, the reference to last year's price, the question written in French.
  • Cases that should go to a person, such as complaints, legal threats or safety issues.
  • Cases where the right answer is "no." If every test case has an answer, you'll never learn whether the agent knows when to decline.

You don't need hundreds to start. Anthropic's guide suggests that "20-50 simple tasks drawn from real failures is a great start." Remove or mask personal information before anything goes into the set.

What to score

  1. Correctness. Are the facts right: prices, hours, lead times, steps?
  2. The right source. Did the answer come from your current price list or policy, not an outdated file or the model's general knowledge? If you're building a central company brain, this is where it earns its keep.
  3. Staying within policy. No refunds, discounts, delivery promises or legal opinions you wouldn't give yourself.
  4. Knowing when to hand off. Does it recognize when a person should take over, and say so, instead of guessing?
  5. Tone. Does it sound like your business and stay polite under pressure?

Some of this can be checked automatically, such as whether the correct price appears. Some needs a person. Another AI model can help grade large sets, but the same guide cautions that such graders "should be closely calibrated with human experts," so a person should keep checking its work.

A worked example: 50 support emails

Say you want an agent to draft replies to support emails. A first eval could look like this:

  1. Pick 50 real support emails from the past year. Keep most routine, but deliberately include awkward ones and several that should have gone to a manager.
  2. For each, have the person who normally answers them note the facts a good reply must contain, which document they come from, anything it must not say, and whether it should be handed off.
  3. Turn the five criteria above into a simple pass/fail rubric.
  4. Run the agent on all 50, several times each, since a case that passes once may fail the next time.
  5. Have the same staff member grade the replies, then go through every failure together.

One case might be a long-time customer asking to return a custom-built item outside the return window. A passing reply is polite, states the policy from the returns document, promises no exception and offers to involve a manager. It fails if it promises a refund, quotes the wrong window or brushes the customer off.

Nobody can predict your score in advance. What you get is a list of specific failures, each pointing to something fixable: a missing document, an unclear policy or instructions that need work.

Set pass bars before launch

Decide what "good enough" means before you see results, so nobody is tempted to move the line afterwards.

The bar can differ by criterion. Policy and hand-off cases should pass every time, because one bad promise can cost real money. For correctness, expect what you'd expect from a trained employee. Tone can start a little lower and improve with use. And a case that passes on some runs but not others isn't passing.

Consider launching in stages: the agent drafts replies for a person to approve, then sends on its own only for request types that clear the bar. At 2MDA, we treat these cases and bars as part of the specification we agree with a client's staff before an agent is built.

Re-run whenever anything changes

An eval isn't a one-time gate. Re-run it whenever the AI model is updated or replaced, the agent's instructions (its "prompt") are edited, your data changes (a new price list, a revised policy) or the agent gets access to a new system.

Any of these can quietly break something that used to work. Anthropic's guide says these regression checks "should have a nearly 100% pass rate," so any drop is a signal to investigate before customers notice.

Keep watching after launch

Your eval covers the situations you thought of. Real customers will find the ones you didn't.

  • Sample real conversations. Each week, have someone grade a handful of the agent's actual exchanges against the same rubric.
  • Add a feedback button. A thumbs up or down for staff and customers flags problems you'd otherwise never hear about. Make sure someone owns it.
  • Feed failures back in. Every real mistake becomes a new test case, so it can't quietly return.

Takeaway

An AI agent can answer the same question differently from one day to the next, so a good demo proves little. Collect real cases from your own history, write down what a good answer looks like, score correctness, sources, policy, hand-offs and tone, and agree the pass bar before launch. Re-run the set whenever anything changes and keep sampling real conversations afterwards. If you can't write the test cases, the job isn't ready to automate yet.