AI agent evaluation is the practice of testing whether an agent does its job properly, measured against cases with known correct outcomes. Evaluation runs offline, before and alongside production, and answers how well the agent performed rather than whether it was permitted to act.

Evaluation sits opposite guardrails in a working system. Guardrails run inline and intervene during a conversation, while evaluation runs against a test set and tells you what to change. Teams that buy on a demo rather than an evaluation tend to find the gap after go-live, once the easy cases are automated and the hard ones are not.

What should AI agent evaluation measure?

A single accuracy number hides everything useful. Agents fail in several distinct ways, and each needs its own measure.

• Task completion: did the case actually close, end to end, with no repeat contact?
  
  

• Correctness: was what the agent said true, and grounded in a source?
  
  

• Tool use: did it call the right systems, with the right parameters, in the right order?
  
  

• Policy adherence: did it follow your rules on disclosure, tone, and what must never be said?
  
  

• Escalation behaviour: did it hand over when it should?
  
  

The last two are where financial services deployments diverge from general benchmarks. An agent can be highly accurate and still unacceptable, because it answered a question it was obliged to escalate.

How do you build an evaluation set?

The best way to evaluate is to use real cases. A set assembled from scripted scenarios measures how the agent handles situations somebody imagined, which is rarely your queue. Where the useful cases come from:

• Your own conversation history, especially the ones that went badly
  
  

• Cases that escalated, which are by definition the ones the easy path missed
  
  

• Complaints and vulnerability cases, weighted more heavily than their volume suggests
  
  

• Known-hard policies, the ones your own team gets wrong
  
  

Size matters less than coverage. A few hundred well-chosen cases spanning the real distribution beats thousands sampled at random, because random sampling reproduces the easy majority you already handle well.

Why do vendor demos prove so little?

A demo is an evaluation the vendor designed. The cases are chosen, the content is prepared, and the failure modes that matter to you are not in scope. Three things a demo cannot show:

• Behaviour when the answer is missing. Demo content covers the demo questions
  
  

• Behaviour under your policies, which the vendor has not implemented
  
  

• Behaviour on your hard cases, which are the reason you are buying at all
  
  

The alternative is straightforward and rarely offered: hand the vendor a set of your own real cases, with the outcomes you would have wanted, and ask them to run it. Our guide on evaluating AI agents in financial services sets out how to construct that.

Frequently asked questions

What is the difference between evaluation and guardrails?

Timing and purpose. Guardrails run inline during a conversation and intervene before a customer sees something, asking whether the agent was permitted to say or do it. Evaluation runs offline against a test set and asks how well it did the job. Production systems need both.

How often should you re-run evaluations?

Whenever anything underneath changes, which is more often than most procurement cycles assume. A vendor changing model, a policy update, or a new case type all invalidate earlier results. Treating evaluation as a one-off at purchase is how deployments drift without anyone noticing.

Can you use an AI model to evaluate an AI agent?

Commonly, but with care. Model-graded evaluation scales to volumes people cannot review, and it inherits the judgment of whatever model is grading. Worth calibrating the grader against human judgment on a sample first, then spot-checking, rather than trusting the score outright.

What does a good evaluation result look like?

A distribution rather than a number. Useful output tells you which case types the agent handles reliably, which it handles inconsistently, and which it fails, so you can scope what it runs on. A single headline percentage is the least actionable form the result can take.