How to Evaluate AI Agents

AI agents are evaluated on whole tasks, not individual answers. That means measuring whether the agent completed the task, whether it took a sensible path to get there, which tools it called and whether those calls were correct, what it cost, and how it behaved when something failed. Evaluating only the final output misses most of the ways agents actually break.

This is the part of agent development that teams most often skip, and it is usually the reason agents that demo well fail in production. It is also one of the first things I look at in any engagement — see what an AI consultant actually does for the wider picture.

Why agents fail differently from chatbots

A chatbot produces an answer. An agent takes a sequence of actions, calls tools, reads results, and decides what to do next. That difference creates failure modes a response-level evaluation cannot see:

• Wrong tool, right-looking answer. The agent produces a plausible output via a tool it should never have called.
• Loops. The agent repeats a step, or oscillates between two, until it hits a step limit or times out.
• Lost context. Information retrieved early is dropped by the time it matters.
• Unsafe actions. The agent takes an irreversible step — sending, deleting, paying — that a human would have paused on.
• Silent degradation. The agent still completes tasks, but takes twice as many steps or costs three times as much as it did last month.

None of these show up if you only grade the final response.

What to measure

A workable agent evaluation covers six dimensions:

Task success is the headline number. The other five explain it — and they are what tell you why a run failed, which is the only thing that lets you fix it.

Build the eval set from real workflows

The most common mistake is inventing test cases. Synthetic tasks measure how well the agent handles situations that never happen.

Build your golden set from real usage instead:

1. Collect actual tasks. From production logs, support tickets, or the workflows your team already does manually.
2. Write a definition of done for each. Not “the answer looks right” — a specific, checkable condition. “The correct invoice ID is extracted and the refund is filed under the right category.”
3. Include the ugly cases. Empty results, ambiguous requests, tools that time out, users who change their mind mid-task.
4. Version the set. When you change the agent, the set stays fixed. Otherwise you are moving the target along with the arrow.

A set of 30–50 well-chosen tasks, drawn from reality and kept stable, is worth more than 500 generated ones.

Evaluate the trajectory, not just the outcome

For each task, record the full trajectory: the sequence of thoughts, tool calls, arguments, results and decisions. Then score it.

Two agents can both succeed, with very different trajectories. One takes three steps and £0.02. The other takes fourteen steps, calls a tool it doesn't need, recovers from a failure it caused itself, and costs £0.40. Both passed. Only one is production-ready.

Practical signals to score on the trajectory:

• Step count against an expected range
• Tool calls — which tools, in what order, with what arguments
• Redundant calls — the same tool called twice with the same input
• Recovery events — failures encountered, and whether they were handled
• Termination — did it stop because it was done, or because it hit a limit?

LLM-as-judge, used carefully

LLM-as-judge is useful for scoring things that are hard to check programmatically — whether a summary is faithful, whether a response is appropriately cautious, whether a plan is sensible.

Two rules make it trustworthy:

Calibrate against human review. Have a person score a sample of the same runs, then compare. If the judge and the human disagree on more than a small fraction, the judge is measuring something other than what you think. Recalibrate before trusting it.

Don't judge with the model you're testing. A model grading its own output has a well-documented bias toward it. Use a different model, or a different prompt structure, and check both against human scores.

For anything that can be checked deterministically — did the right ID get extracted, was the correct tool called, is the total arithmetic correct — use code, not a judge. Judges are for the parts you cannot check mechanically.

Evaluating tool calls, including MCP tools

If your agent uses MCP servers or external tools, each tool call is a testable event. For a given task, you know which tools should be called and roughly with what arguments. Score:

• Was the right tool selected?
• Were the arguments correct and complete?
• Was the result actually used, or fetched and ignored?
• Were permissions respected — no calls outside the agent's scope?

This is often where the cheapest, highest-signal failures show up. A tool called with a malformed argument is a bug you can fix in an afternoon once you can see it.

Evaluating agent skills

If your agent loads reusable skills — instructions, scripts or tools it pulls in on demand — each skill needs its own evaluation. A skill only helps if the agent triggers it at the right time and follows it correctly.

Test each skill on three questions:

1. Triggering accuracy. Does the agent load the skill when it should, and not load it when it shouldn't? Both directions matter — a skill that fires too eagerly is as damaging as one that never fires.
2. With-skill vs without-skill. Run the same task set with the skill enabled and disabled. If scores don't improve, the skill isn't earning its place in the context window.
3. Consistency. Run the same task several times. A skill that works two times in five is not working.

Skills that fail all three should be rewritten, merged or retired. A large skill library with no evaluation is a liability, not a capability. If you need help setting this up, see agent skills evaluation.

Put regression testing in CI

Agent evaluation is only useful if it runs automatically. Wire your task set into CI so that every change to prompts, tools, models or orchestration runs the same evaluations and reports the same metrics.

The practical setup:

• Fixed task set, versioned, run on every meaningful change
• Automated scoring where possible, sampled human review where not
• Thresholds that fail the build — e.g. task success dropping more than 5%, or cost per task rising more than 20%
• Tracing (LangFuse, or equivalent) so a failing run can be inspected end to end

Without this, evaluation is a one-off exercise that tells you how the agent behaved on the day you ran it.

Red-teaming and safety

Add a separate set of adversarial tasks: prompts designed to make the agent exceed its permissions, leak data, take an irreversible action, or follow an instruction it should refuse.

This is not the same as accuracy testing. An agent can score well on task success and still be unsafe. Run these separately, and treat any failure here as a blocker rather than a metric.

Common mistakes

• Evaluating only the final answer. Misses loops, wrong tools, wasted cost and unsafe actions.
• Synthetic test sets. Measure the agent on situations that don't occur.
• Judging with the same model. Systematic bias toward its own output.
• No human calibration. The judge drifts and no one notices.
• No regression suite. Every change is a fresh unknown.
• Tracking success only. An agent that gets slower and more expensive each month still “passes”.
• One run per task. Agents are stochastic. Score across multiple runs, or you are measuring luck.

A minimum viable eval setup

Before shipping an agent to real users, any team should have at least:

• A versioned golden set of 30–50 real tasks, each with a checkable definition of done
• Full trajectory tracing for every evaluation run — tool calls, arguments, results, step count
• Task success, cost and step count tracked per run and compared against the previous version
• An LLM judge calibrated against human review, used only where code cannot check the result
• A red-team set for permissions, data leakage and irreversible actions, treated as a release blocker
• All of it running in CI, with thresholds that fail the build

The short version

Define the task clearly, score the trajectory as well as the outcome, build the test set from real usage, calibrate any LLM judge against human review, and run the whole thing in CI on every change. That is the difference between an agent that demos well and one that survives production.

Yuliya Halavachova is an AI Engineering Manager at a global enterprise software company, leading a team of 15 engineers shipping AI agents in a B2B SaaS product. She provides AI consulting on agent and LLM evaluation for UK companies.