The situation
A subscriber-facing refund agent has been live for six weeks. It runs on Amazon Bedrock AgentCore and handles a narrow job. A subscriber asks about a refund; the agent looks up the subscription, pulls the delivery record for the week in question, and checks the refund policy in a knowledge base. It then either issues a credit through a payments tool or explains why it cannot.
Support has been sampling its work. Roughly eight replies in ten are the answer a human would have given. The other two are wrong in ways nobody has managed to characterise. The ticket queue has a label for them, and the label is “agent got it wrong”, which is where the analysis stops.
The failures are not all the same failure. In some the agent called the wrong tool, checking the delivery record when the subscription’s pause history was what the task needed. In some it called the right tool with a mangled argument, a delivery date off by a week. In some a tool returned a 500 and the run carried on with the error body in place of a result, no retry attempted. In at least one the reply was correct while the run underneath it had already issued a credit through the payments tool. The right answer arrived with a side effect nobody wanted. A single number for answer correctness gives all four the same score, and the team is being asked to fix something that number cannot locate.
What actually matters
An agent emits more than an output. It emits a trajectory: a sequence of steps in which the model selects the next action, a tool runs, the result goes back into the context, and the model selects again, until the task reaches its end state or the run stops short of one. The final string is one artefact of that sequence, and it is the artefact furthest from the steps that produced it. Scoring only the string gives a pass or fail with no attribution, which is the state this team is in. Every question they want answered is a question about a step. Was the tool the right one. Was the argument correct. Was a failed call retried. Those are visible only if the evaluation reads the trajectory.
The two errors that a final-answer score cannot reach are the ones that hurt most. An agent can be right by the wrong route: the reply is correct, but the run took eleven steps instead of four, or reached the answer through a tool that also moved money. An agent can also be wrong under a fluent summary, where a tool call failed without the failure reaching the reply, the model produced a plausible-looking value in its place, and the text reads exactly like the successful runs. Both look identical to answer correctness. Both are obvious the moment you assert on the steps. The first is the case for evaluating agents at all rather than treating them as an opaque question-answering box. A correct answer produced by an unwanted side effect is a production incident with a passing test beside it.
The run is also non-deterministic. The same scripted task, run twice with the same inputs, can take different routes, use a different number of turns, and land in different places. One pass through a task is a sample, not a measurement, and treating it as a regression test produces a suite that fails on Tuesday and passes on Wednesday with nothing changed. The unit that means something is a pass rate over N runs of the same scripted task: run each task twenty or fifty times, count completions, and compare rates rather than individual runs. That reframes the whole harness. A quality gate becomes “completion rate on the refund suite is at least 92%”, not “the refund test passed”, and a regression becomes a rate that moved outside its usual band rather than a single red result.
Running the evaluation at all is harder for agents than for retrieval or generation, because an agent’s tools do real work. Evaluating a refund agent by letting it run means issuing refunds, sending emails and writing rows, dozens of times per suite execution, on every pipeline run. Tool isolation is therefore part of the harness design rather than an afterthought. Side-effecting tools get replaced by a mocked gateway that returns scripted responses and records what it was called with, or sandboxed against a throwaway account whose state is reset between runs. A mocked gateway also removes the variation in the tool layer. When the tool always returns the same delivery record, whatever variation is left came from the model, and that is the variation being measured.
What we’ll filter on
- Trajectory visibility: does it read the intermediate steps, or only the prompt and the final answer?
- Task completion: does it score whether the task was finished, as distinct from whether the text was good?
- Labelling effort: does it need a golden trajectory for every task, or just a golden outcome?
- Unattended operation: can it run in a deployment pipeline as a quality gate, without a human in the loop?
- Side-effect isolation: does the approach give you somewhere to put mocked or sandboxed tools?
The landscape
Amazon Bedrock model evaluation jobs
Bedrock’s evaluation jobs are built around prompt in, answer out. A custom dataset is JSON Lines in S3, up to 1,000 records per job, each record a prompt with an optional referenceResponse and category. You pick automatic metrics or a judge model, and get scored outputs back. For comparing two foundation models on a summarisation task that shape fits, and it is the tool this team already has wired up.
Intermediate steps are not in that format. There is nowhere to record that the agent called the delivery tool with the wrong date, because the record has no field for a tool call. Pointed at the refund agent it reports that 80% of replies are acceptable, which is the number the team started with.
Amazon Bedrock AgentCore Evaluations
AgentCore Evaluations is the agent-shaped service. It reads the telemetry an instrumented agent already emits, in either the OpenTelemetry generative-AI or the OpenInference semantic conventions, and scores it with built-in or custom evaluators that run as judge-model calls. The built-in set works at three levels. Session evaluators cover goal success rate, with a ground-truth variant. Trace evaluators cover correctness, faithfulness, coherence and instruction following, among others. Tool-level evaluators cover tool selection accuracy and tool parameter accuracy, which is most of what a plain output evaluation cannot reach.
It runs three ways: online against a sampled share of live sessions, on demand against spans or traces you name, and as a batch job over sessions stored in CloudWatch Logs. Batch evaluation takes ground truth from session metadata, including expected responses, assertions and expected tool trajectories. A dataset runner that invokes the agent across a set of scenarios and evaluates the results in one call is in public preview at the time of writing, so its APIs may still move.
The trade-offs are the metrics and the plumbing. Built-in evaluator prompt templates cannot be modified, so a domain-specific rule such as “the payments tool must never be called when the delivery record shows a completed refund” needs a custom evaluator or a harness of your own. The agent also has to be instrumented under a scope name the service recognises and exporting to CloudWatch, or there are no spans to score.
AgentCore observability traces plus your own judge
An instrumented agent already emits spans for every model call and every tool call, carrying the tool name, the arguments, the result, timings and token counts. That trace is a complete record of the trajectory, and it exists whether or not anyone evaluates it. Running your own judge over it turns tool calling observability into a score: give the judge the task, the recorded steps and a scoring guide, and ask it to rate whether the path from step to step holds together.
This is where reasoning quality assessment in multi-step workflows lands, when the failure is not a wrong tool or a bad argument but an incoherent plan. It carries the usual judge caveats: the rubric needs calibrating against human ratings, the judge is another non-deterministic component, and the scores move when the judge model changes. AgentCore Evaluations has custom evaluators for the same job, so run your own only where the rubric or the judge model you need sits outside what the service offers. Either way it scores production traces and not only scripted tasks.
A hand-built harness of scripted tasks
The harness is a set of tasks written as code. Each task fixes an input, points the agent at a mocked gateway or a sandboxed tool set, and runs it N times. It then asserts over the recorded tool calls: this tool was called, these tools were not, the date argument matched the subscription’s delivery window, the run finished within six steps, the payments tool was called exactly once with this amount.
Nothing else gives you assertions that specific, and nothing else runs as cleanly in a pipeline, because the output is a pass rate and a set of named failures rather than a score somebody has to interpret. You write and maintain it, and its assertions encode a route. Tighten them too far and the suite fails every time the agent finds a legitimate second way to do the job; leave them loose and they stop catching the right-answer-wrong-route case they were built for.
Human review of a stratified sample
Someone reads traces. Not all of them, and not a random draw either. The sample is stratified, weighted toward the runs the automated metrics rate as marginal, the ones that took an unusual number of steps, and the ones where a tool errored. Human review is slow and does not gate a deployment, and it is the only thing on this list that finds failure modes nobody thought to assert on. It is also how the judge rubric gets calibrated in the first place.
Evaluation
Side by side
| Approach | Reads the trajectory | Scores task completion | Needs golden trajectories | Runs unattended | Isolates side effects |
|---|---|---|---|---|---|
| Bedrock model evaluation jobs | ✗ | ✗ | ✗ (golden outcomes) | ✓ | ✗ |
| AgentCore Evaluations | ✓ | ✓ | ✗ (ground truth optional) | ✓ | ✗ (you supply the tools) |
| AgentCore traces + your own judge | ✓ | ✓ | ✗ (rubric, not labels) | ✓ | n/a (scores runs after the fact) |
| Scripted-task harness | ✓ | ✓ | ✓ (assertions encode the route) | ✓ | ✓ |
| Stratified human review | ✓ | ✓ | ✗ | ✗ | n/a |
Read the first column downward. Only one row misses the trajectory, and it is the row this team is currently using, which explains the shape of their problem better than anything else in the table. Read the last two columns across and the split is between the approaches that gate a pipeline and the one that finds what the gates missed. Nothing here is a single answer. The harness and AgentCore Evaluations gate deployments, a judge scores the reasoning the assertions cannot express, and human review keeps the other three pointed at real failures.
One run, one assertion per step
The solution
Build the agent performance framework around a named metric set rather than a single score, because each metric answers a different one of the questions support could not answer. Seven are worth collecting.
Task completion rate is the fraction of runs that reached the task’s defined end state, measured over N runs of each scripted task. Gate on this one first. A route metric computed on a task the agent never finished measures nothing: tool selection accuracy of 100% across the four steps of a run that then stalled is a number that looks reassuring and describes a failure.
Tool selection accuracy is the fraction of steps where the tool the agent chose was one the task legitimately needed, plus the mirror of it, the fraction of runs where a forbidden tool was never called. The forbidden half is what catches the right-answer-with-a-side-effect case, and it is worth asserting explicitly on every side-effecting tool in the gateway’s schema. AgentCore Evaluations ships a tool selection accuracy evaluator, so that half needs no code of your own; the forbidden-tool assertion is yours to write.
Tool-argument validity is the fraction of tool calls whose arguments were well-formed and correct against the fixture: dates inside the subscription’s window, identifiers that resolve, enumerated values from the schema. The managed equivalent is the tool parameter accuracy evaluator. This is where the delivery-date-off-by-a-week failure lands, and it is invisible to everything except an assertion on the recorded call.
Steps to completion and turns measure efficiency: how many tool calls the run needed, and how many exchanges with the subscriber it took. Both need a budget and an alert when a release moves them. A rising step count means more calls to reach the same end state, and it shows up in the bill before it shows up in the quality score.
Recovery rate is the fraction of runs containing a failed tool call that went on to complete the task anyway. Without it a tool error is indistinguishable from a tool result in the metrics, which is precisely the failure mode in the ticket queue: a 500 read as “no deliveries found” and never retried.
Cost and latency per completed task are the operational pair, and the denominator matters. Cost per invocation looks better for an agent that abandons tasks early; cost per completed task does not. Both come off the same spans that carry the token counts.
A judged reasoning-quality score covers multi-step workflows where the plan itself is the defect. Feed the recorded trajectory to a judge with a rubric that rates whether each step followed from the last and whether the agent used what the previous tool returned. Calibrate the rubric against human ratings on a sample before trusting the score, and re-calibrate when the judge model changes.
Getting those numbers needs three pieces of plumbing. The first is the harness: scripted tasks that fix inputs, a mocked gateway standing in for every side-effecting tool, and N repeated runs per task so the output is a rate. Wire it into the deployment pipeline as a gate on completion rate and a warning band on the route metrics; a release that holds completion steady while step count climbs is a release worth looking at.
The second is tool calling observability in production, where the harness cannot follow. Emit a span per tool call with the tool name, argument shape, result status and duration. Roll those up into call pattern tracking: which tools get called, in what order, how often, and how that distribution moves week to week. Performance metric collection at the tool level gives you per-tool error rates and latencies, and usage baselines for anomaly detection turn the normal distribution of calls into an alarm when it shifts. A doubling in payments-tool calls shows up there before any subscriber complains. Where several agents cooperate, the same spans carry multi-agent coordination tracking, so a handoff that stalls is attributable to the agent that dropped it rather than to the system as a whole.
The third is managed evaluation alongside the hand-built parts. AgentCore Evaluations scores goal success and tool usage without a harness to maintain, which gives a trend across the whole agent once the telemetry is in place, while the scripted suite carries the assertions specific to this domain. Bedrock’s model evaluation jobs stay useful for what they were built for, comparing candidate foundation models on the underlying generation quality, and they go on measuring answers rather than runs.
Worked example
Take the three uncharacterised failures from the ticket queue and run each through the framework.
The wrong tool
A subscriber asks about a refund for a week they had paused. The agent calls getDeliveries first, gets an empty result for that week, and replies that no delivery was made so no refund applies. The answer is wrong: a paused week is refundable under the policy, and the pause history lives on the subscription, not the delivery record.
Answer correctness marks this one wrong and stops. Tool selection accuracy marks step one as a wrong choice, and the assertion “the first call is getSubscription” fails by name. Across fifty runs of the task the failure appears in eleven. That turns a sampled anecdote into a 78% completion rate on one task, and points at the tool description in the gateway schema as the thing to change.
The bad argument
The agent picks the right tools in the right order and calls getDeliveries with week=2026-07-06 when the subscriber asked about the week of the thirteenth. The reply is fluent, cites a real delivery, and refunds nothing. Every step passed tool selection. The run completed. A judge scoring the reasoning path finds it coherent, because it is coherent, just about the wrong week.
Tool-argument validity is the only metric that catches this, and it catches it because the harness fixture records which week the task was about and the assertion compares the argument against it. That is the case for scripted tasks over production traces: production has no ground truth for what the argument should have been.
The unretried error
getDeliveries returns a 500. The error body goes back into the context as the tool result, and the model produces a fluent reply saying there were no deliveries. Completion rate counts the run as complete, because a reply was produced. Tool selection and argument validity both pass. Only recovery rate catches it: a run containing a failed tool call that never retried, and a step-level assertion that a non-200 must be followed by a retry or an explicit failure to the subscriber.
In production, where the harness is absent, the same failure surfaces through per-tool error rates in performance metric collection and a usage baseline that alarms when getDeliveries errors at ten times its normal rate for an hour. The evaluation harness names the defect; the observability baseline catches the next occurrence at three in the morning.
What’s worth remembering
- An agent emits a trajectory, so a single answer-correctness score cannot attribute a failure to a wrong tool, a bad argument, an unretried error, or an unwanted side effect.
- Agent runs are non-deterministic, so the unit that means anything is a pass rate over N runs of the same scripted task, not the result of one pass.
- Gate on task completion rate first: tool selection accuracy and the other route metrics describe nothing when computed over a task the agent never finished.
- Collect the metric set by name (task completion rate, tool selection accuracy, tool-argument validity, steps to completion and turns, recovery rate, cost and latency per completed task, judged reasoning quality) rather than compressing it into one number.
- Bedrock model evaluation jobs score prompt in and answer out; AgentCore Evaluations scores the trajectory, with built-in evaluators for goal success, tool selection accuracy and tool parameter accuracy; a judge over the traces, managed or your own, is what reaches reasoning quality in multi-step workflows.
- Any evaluation that calls real tools spends real money and moves real state, so a mocked gateway or a sandboxed tool set is part of the harness design, not a refinement to add later.