How do AI agent evaluations catch bad answers before customers do?

Test case: a customer asking for a discount

What each test case checks

• Multi-message conversationsA case can include several customer messages, plus earlier history the agent "remembers".
• Score and reasoningAn AI judge scores the reply against the criteria you wrote and explains what worked and what did not.
• Must and must notPhrases that must appear in the reply, and phrases that must not, such as a promised discount or a price that does not exist.
• Tool useCheck that the agent called the right tool, with the right details, and that the tool did not return an error.
• MemoryCheck that the agent saved, or did not save, what it should about the contact.
• RepetitionsRun each case several times to see that the agent is consistent and not right just once.

Tests that never touch customers

• Mocked toolsEach tool gets a prepared answer: text, JSON, an error or a sequence of answers. Your real calendar, CRM and payments stay untouched.
• Block unlisted toolsOptionally make any tool without a mock return an error, to see how the agent copes.
• Cases from real conversationsA conversation the agent had with a customer becomes a test case, with suggested mocks based on the tools it used.
• Case writing assistantAn AI assistant suggests new cases, varies them, and tightens existing ones so they are harder to pass.

After the agent goes live

• Scheduled runsA test suite runs on a schedule and sends an alert when it fails.
• Score historyAn overall score and a score per test category over time, so you can see whether an instructions change helped or hurt.
• Pass thresholdSet the score at which a suite counts as passed.
• Autonomous agents tooTest a work round of an autonomous agent: which tools it used, how many actions it took, whether it updated a goal and whether it asked for approval when it should.

Start testing

1. 1Create a test suite

   For an existing agent, or from a real conversation you want to reproduce.

2. 2Write test cases

   What the customer writes, what must happen, what must not, and how the judge should score it.

3. 3Set up mocked tools

   A prepared answer for every tool the agent might call.

4. 4Run it and schedule it

   Run it, read the reasoning, fix the instructions, and schedule a regular run.

Questions about agent evaluations