AI Agent Evals: How We Test an Agent Before Launch
Ask a vendor how they know their agent works and most will show you a demo. A demo is one scenario, chosen by the person who built it. An eval is two hundred scenarios, chosen from what your users actually do, graded automatically, run on every change. It is the difference between an agent you hope works and one you know works.
- Published
- Updated
Build the scenario set from reality
The eval set is the product. Source it from real transcripts, support tickets and the tasks people actually attempt — not from what the team imagines. Aim for 150–300 scenarios on a first pass, weighted toward the cases where a mistake is expensive.
- Happy paths: the ten most common tasks, several phrasings each.
- Edge cases: missing information, contradictory instructions, out-of-scope requests.
- Adversarial: prompt injection inside a document or tool result, attempts to extract secrets.
- Multi-step: tasks that need three or more tool calls in the right order.
Choose the grader per scenario
| Method | Use for | Cost | Reliability |
|---|---|---|---|
| Exact / structured match | Tool called with right arguments, correct final state | Free | Highest |
| Rubric via a model | Tone, completeness, correctness of prose | Cents per case | High if rubric is concrete |
| Pairwise comparison | Choosing between two agent versions | Cents per case | Good for relative calls |
| Human review | Sampling model grades; ambiguous cases | Expensive | Ground truth |
Wherever a tool call or database state can be checked, check it directly. Save model graders for judgement calls, and audit a sample of their grades by hand each week.
Set the bar by cost of error
A 92% pass rate is excellent for an internal research agent and unacceptable for one that issues refunds. Set the bar per scenario class, gate the risky classes behind human approval until they pass, and treat every regression as a blocking bug. Keep a held-out set the team never tunes against, so you can trust the number.
How we run it
Every agent we ship carries its eval suite in the repo, running in CI. We build suites for agents other teams wrote, and we teach the method in a one-day workshop so your engineers can keep it alive as the model, the tools and the business change.
Frequently asked questions
How many scenarios are enough?
Enough that the pass rate stops moving when you add more — usually 150–300 for a focused agent. Breadth across failure types matters more than raw count.
How much does an eval suite cost to build?
Typically 15–25% of the agent build, or $8k–$30k standalone for an existing agent. Running it costs cents to a few dollars per full pass.
Do evals go stale?
Yes. Models change, tools change, users change. Budget a few engineer-hours a month to add fresh scenarios from production and prune obsolete ones.
