A practical approach to evaluating agent outcomes, trajectories, tools, and safety boundaries before and after release.
A good final answer can hide a bad trajectory
Evaluating a chatbot often starts with its response. An agent has a larger failure surface: it interprets a goal, chooses tools, supplies arguments, observes results, updates a plan, and decides when to stop. It may eventually produce the correct answer after reading prohibited data, repeating an expensive call, or making and reversing an unsafe change. Outcome-only scoring would mark that run as a success.
Agent evaluation therefore needs two views. Outcome evaluation asks whether the requested state was reached with a useful result. Trajectory evaluation asks whether the path was authorized, efficient, grounded in observations, and recoverable. The relative importance depends on the domain, but consequential tools make trajectory quality impossible to ignore.
Build a suite shaped by real work and real risk
A useful evaluation set represents common tasks, difficult edge cases, and failures with disproportionate consequences. Derive cases from workflow observation, historical incidents, operator interviews, and explicit policy. Include incomplete requests, conflicting instructions, unavailable tools, stale observations, permission changes, malicious content in retrieved data, and situations where the right behavior is to ask or stop.
- Task success: did the environment reach the intended state, not merely receive a confident summary?
- Tool correctness: were the right operations called with valid, minimal, and authorized arguments?
- Grounding: did decisions follow tool observations, or did the model invent state between calls?
- Efficiency: did the run avoid loops, redundant retrieval, and unnecessary high-cost models?
- Safety: did the system respect identity, data, financial, and external-action boundaries?
- Recovery: after a timeout or tool error, was partial state left understandable and safe to continue?
Use deterministic checks before subjective graders
Many agent properties can be checked directly. Did a forbidden tool run? Did a write occur before approval? Does the created record match the expected state? Were repeated operations idempotent? Deterministic checks are cheap, explainable, and stable. Use them wherever the environment exposes a fact.
Human or model graders are useful for qualities such as clarity, sufficiency, and whether a plan was sensible. Give graders a rubric with observable anchors and an option for “cannot determine.” Calibrate model graders against independent human labels and revisit disagreements. A sophisticated judge prompt cannot repair an ambiguous definition of success.
Put each guardrail at the layer that can enforce it
- Constrain interfaces
Give tools narrow schemas, typed values, allow-listed destinations, and explicit tenant and actor context.
- Authorize at execution time
Check current permissions inside the service performing the action; never treat the model’s request as authorization.
- Gate consequential actions
Show a concrete preview and require approval for external messages, money movement, access changes, deletion, or irreversible updates.
- Limit resources and loops
Enforce budgets for calls, tokens, time, repeated actions, and total cost outside the model’s own reasoning.
- Validate resulting state
After a tool reports success, confirm the expected state where feasible and return discrepancies to a safe recovery path.
Prompts still matter: they define the role, available evidence, escalation behavior, and expected explanation. But prompts are guidance, not a security boundary. A robust design assumes the planner can misunderstand instructions or encounter adversarial text and ensures that the surrounding system limits the damage.
Turn production traces into controlled learning
Before release, run the suite on every meaningful change to prompts, models, tools, retrieval, or policies. Compare candidate and baseline at the case level, investigate regressions, and retain representative traces. In production, monitor hard policy events, tool error patterns, approval rejection, loop termination, latency, and cost. Sampling successful runs matters too, because silent inefficiency rarely generates a support ticket.
User feedback is evidence, not an automatic label. A rejected draft may be wrong, stylistically different, or blocked by context the system never received. Review feedback, connect it to a trace, remove sensitive material, and promote stable examples into the offline suite. Evaluation works when it is a release discipline and an operational habit, not a benchmark produced once for launch approval.