The situation
A subscriber-facing refund agent has been live for six weeks. It runs on Amazon Bedrock AgentCore and handles a narrow job. A subscriber asks about a refund; the agent looks up the subscription, pulls the delivery record for the week in question, and checks the refund policy in a knowledge base. It then either issues a credit through a payments tool or explains why it cannot.
Support has been sampling its work. Roughly eight replies in ten are the answer a human would have given. The other two are wrong in ways nobody has managed to characterise. The ticket queue has a label for them, and the label is “agent got it wrong”, which is where the analysis stops.
The failures are not all the same failure. In some the agent called the wrong tool, checking the delivery record when it needed the subscription’s pause history. In some it called the right tool with a mangled argument, a delivery date off by a week. In some a tool returned a 500, the agent read the error as a negative result and carried on, never retrying. And in at least one the reply was correct while the run underneath it had already issued a credit through the payments tool, meaning the right answer arrived alongside a side effect nobody wanted. A single number for answer correctness covers all four of those with the same score, and the team is being asked to fix something that number cannot locate.
What actually matters
An agent does not emit an output. It emits a trajectory: a sequence of steps in which a model decides what to do next, calls a tool, reads what comes back, and decides again, until it either finishes the task or gives up. The final string is one artefact of that sequence, and it is the artefact furthest from the decisions that produced it. Scoring only the string gives you a pass or fail with no attribution, which is the state this team is in. Every question they actually want answered is a question about a step. Did it pick the right tool. Did the argument make sense. Did it recover when the tool broke. Steps are only visible if the evaluation reads the trajectory.
The two errors that a final-answer score cannot see are the ones that hurt most. An agent can be right by the wrong route: the reply is correct, but the run took eleven steps instead of four, or reached the answer through a tool that also moved money. An agent can also be wrong with a confident summary, where a tool call failed silently, the model inferred a plausible-looking value, and the reply reads exactly like the successful ones. Both look identical to answer correctness. Both are obvious the moment you assert on the steps. And the first of them is the case for evaluating agents at all rather than treating them as an opaque question-answering box. A correct answer produced by an unwanted side effect is a production incident with a passing test beside it.
The run is also non-deterministic. The same scripted task, run twice with the same inputs, can take different routes, use a different number of turns, and land in different places. One pass through a task is a sample, not a measurement, and treating it as a regression test produces a suite that fails on Tuesday and passes on Wednesday with nothing changed. The unit that means something is a pass rate over N runs of the same scripted task: run each task twenty or fifty times, count completions, and compare rates rather than individual runs. That reframes the whole harness. A quality gate becomes “completion rate on the refund suite is at least 92%”, not “the refund test passed”, and a regression becomes a rate that moved outside its usual band rather than a single red result.
Then there is the cost of running the evaluation, which for agents is unlike anything in retrieval or generation evaluation. An agent’s tools do real work. Evaluating a refund agent by letting it run means issuing refunds, sending emails, and writing rows, dozens of times per suite execution, on every pipeline run. So tool isolation is part of the harness design rather than an afterthought. Side-effecting tools get replaced by a mocked gateway that returns scripted responses and records what it was asked for, or sandboxed against a throwaway account whose state is reset between runs. A mocked gateway gives you determinism in the tool layer as well. When the tool always returns the same delivery record, any variation left in the run came from the model, and that is the variation you are trying to measure.
What we’ll filter on
- Trajectory visibility: does it see the intermediate steps, or only the prompt and the final answer?
- Task completion: does it score whether the task was finished, as distinct from whether the text was good?
- Labelling cost: does it need a golden trajectory for every task, or just a golden outcome?
- Unattended operation: can it run in a deployment pipeline as a quality gate, without a human in the loop?
- Side-effect isolation: does the approach give you somewhere to put mocked or sandboxed tools?
The landscape
Amazon Bedrock Model Evaluations
Bedrock’s evaluation jobs are built around prompt in, answer out. You supply a dataset of prompts with optional reference responses, pick automatic metrics or a judge model, and get scored outputs back. For comparing two foundation models on a summarisation task that shape is exactly right, and it is the tool this team already has wired up.
It is blind to intermediate steps. There is no place in a prompt-and-response dataset to record that the agent called the delivery tool with the wrong date, because the dataset format has no concept of a tool call. Pointed at the refund agent it will faithfully tell you that 80% of replies are acceptable, which is the number the team started with.
Amazon Bedrock Agent evaluations
Bedrock Agent evaluations are the agent-shaped member of the same family. Instead of scoring a response in isolation they score an agent invocation. The trajectory is part of what gets scored: whether the task was completed, whether the tools it chose were the ones the task needed, whether the intermediate reasoning holds together. That covers the two questions a plain output evaluation cannot reach, task completion and tool usage, and it does so as a managed job rather than a harness somebody has to maintain.
The trade-off is that you are working inside the metrics and the invocation shape the service provides. A domain-specific assertion, such as “the payments tool must never be called when the delivery record shows a completed refund”, is not something a general agent evaluation will express for you.
AgentCore observability traces plus a judge
An instrumented agent already emits spans for every model call and every tool call, with arguments, results, timings and token counts on them. That trace is a complete record of the trajectory, and it exists whether or not anyone evaluates it. Feeding a trace to a judge model with a rubric turns tool calling observability into a score. Give the judge the task, the recorded steps and a scoring guide, and ask it to rate whether the reasoning path was sound.
This is the only option in the landscape that gets at reasoning quality assessment in multi-step workflows, where the failure is not a wrong tool or a bad argument but an incoherent plan. It carries the usual judge caveats: the rubric needs calibrating against human ratings, the judge is another non-deterministic component, and its scores drift when the judge model changes. It also works on production traces, not only on scripted tasks, which makes it the one approach that can score runs that really happened.
A hand-built harness of scripted tasks
The harness is a set of tasks written as code. Each task fixes an input, points the agent at a mocked gateway or a sandboxed tool set, and runs it N times. It then asserts over the recorded tool calls: this tool was called, these tools were not, the date argument matched the subscription’s delivery window, the run finished within six steps, the payments tool was called exactly once with this amount.
Nothing else gives you assertions that specific, and nothing else runs as cleanly in a pipeline, because the output is a pass rate and a set of named failures rather than a score somebody has to interpret. The cost is that you write and maintain it, and that its assertions encode a route. Tighten them too far and the suite fails every time the agent finds a legitimate second way to do the job; leave them loose and they stop catching the right-answer-wrong-route case they were built for.
Human review of a stratified sample
Someone reads traces. Not all of them, and not a random draw either. The sample is stratified, weighted toward the runs the automated metrics rate as marginal, the ones that took an unusual number of steps, and the ones where a tool errored. Human review is slow and does not gate a deployment, and it is the only thing on this list that finds failure modes nobody thought to assert on. It is also how the judge rubric gets calibrated in the first place.
Evaluation
Side by side
| Approach | Sees the trajectory | Scores task completion | Needs golden trajectories | Runs unattended | Isolates side effects |
|---|---|---|---|---|---|
| Bedrock Model Evaluations | ✗ | ✗ | ✗ (golden outcomes) | ✓ | ✗ |
| Bedrock Agent evaluations | ✓ | ✓ | ✗ (golden outcomes) | ✓ | ✗ (you supply the tools) |
| AgentCore traces + judge | ✓ | ✓ | ✗ (rubric, not labels) | ✓ | n/a (scores runs after the fact) |
| Scripted-task harness | ✓ | ✓ | ✓ (assertions encode the route) | ✓ | ✓ |
| Stratified human review | ✓ | ✓ | ✗ | ✗ | n/a |
Read the first column downward. Only one row is blind to the trajectory, and it is the row this team is currently using, which explains the shape of their problem better than anything else in the table. Read the last two columns across and the split is between the approaches that gate a pipeline and the one that finds what the gates missed. Nothing here is a single answer; the harness and the agent evaluation gate deployments, the judge scores the reasoning the assertions cannot express, and human review keeps the other three pointed at real failures.
One run, one assertion per step
The solution
Build the agent performance framework around a named metric set rather than a single score, because each metric answers a different one of the questions support could not answer. Seven are worth collecting.
Task completion rate is the fraction of runs that reached the task’s defined end state, measured over N runs of each scripted task. Gate on this one first. A route metric computed on a task the agent never finished measures nothing: tool selection accuracy of 100% across the four steps of a run that then stalled is a number that looks reassuring and describes a failure.
Tool selection accuracy is the fraction of steps where the tool the agent chose was one the task legitimately needed, plus the mirror of it, the fraction of runs where a forbidden tool was never touched. The forbidden half is what catches the right-answer-with-a-side-effect case, and it is worth asserting explicitly on every side-effecting tool in the gateway’s schema.
Tool-argument validity is the fraction of tool calls whose arguments were well-formed and correct against the fixture: dates inside the subscription’s window, identifiers that resolve, enumerated values from the schema. This is where the delivery-date-off-by-a-week failure lands, and it is invisible to everything except an assertion on the recorded call.
Steps to completion and turns measure efficiency: how many tool calls the run needed, and how many exchanges with the subscriber it took. Both want a budget and an alert when a release moves them. A rising step count usually means the agent is groping rather than deciding, and it shows up in the bill before it shows up in the quality score.
Recovery rate is the fraction of runs containing a failed tool call that went on to complete the task anyway. Without it a tool error is indistinguishable from a tool result in the metrics, which is precisely the failure mode in the ticket queue: a 500 read as “no deliveries found” and never retried.
Cost and latency per completed task are the operational pair, and the denominator matters. Cost per invocation flatters an agent that gives up early; cost per completed task does not. Both come off the same spans that carry the token counts.
A judged reasoning-quality score covers multi-step workflows where the plan itself is the defect. Feed the recorded trajectory to a judge with a rubric that rates whether each step followed from the last and whether the agent used what the previous tool returned. Calibrate the rubric against human ratings on a sample before trusting the score, and re-calibrate when the judge model changes.
Getting those numbers needs three pieces of plumbing. The first is the harness: scripted tasks that fix inputs, a mocked gateway standing in for every side-effecting tool, and N repeated runs per task so the output is a rate. Wire it into the deployment pipeline as a gate on completion rate and a warning band on the route metrics; a release that holds completion steady while step count climbs is a release worth looking at.
The second is tool calling observability in production, where the harness cannot follow. Emit a span per tool call with the tool name, argument shape, result status and duration. Roll those up into call pattern tracking: which tools get called, in what order, how often, and how that distribution moves week to week. Performance metric collection at the tool level gives you per-tool error rates and latencies, and usage baselines for anomaly detection turn the normal distribution of calls into an alarm when it shifts. An agent that starts calling the payments tool twice as often is telling you something before any subscriber complains. Where several agents cooperate, the same spans carry multi-agent coordination tracking, so a handoff that stalls is attributable to the agent that dropped it rather than to the system as a whole.
The third is managed evaluation alongside the hand-built parts. Amazon Bedrock Agent evaluations score task completion and tool usage effectiveness without a harness to maintain. That makes them the cheap way to get a trend on the whole agent, while the scripted suite carries the assertions specific to this domain. Amazon Bedrock Model Evaluations stays useful for what it was built for, comparing candidate foundation models on the underlying generation quality, and it goes on measuring answers rather than runs.
Worked example
Take the three uncharacterised failures from the ticket queue and run each through the framework.
The wrong tool
A subscriber asks about a refund for a week they had paused. The agent calls getDeliveries first, gets an empty result for that week, and replies that no delivery was made so no refund applies. The answer is wrong: a paused week is refundable under the policy, and the pause history lives on the subscription, not the delivery record.
Answer correctness marks this one wrong and stops. Tool selection accuracy marks step one as a wrong choice, and the assertion “the first call is getSubscription” fails by name. Across fifty runs of the task the failure appears in eleven. That turns a sampled anecdote into a 78% completion rate on one task, and points at the tool description in the gateway schema as the thing to change.
The bad argument
The agent picks the right tools in the right order and calls getDeliveries with week=2026-07-06 when the subscriber asked about the week of the thirteenth. The reply is fluent, cites a real delivery, and refunds nothing. Every step passed tool selection. The run completed. A judge scoring the reasoning path finds it coherent, because it is coherent, just about the wrong week.
Tool-argument validity is the only metric that catches this, and it catches it because the harness fixture knows which week the task was about and asserts the argument against it. That is the argument for scripted tasks over production traces: production has no ground truth for what the argument should have been.
The unretried error
getDeliveries returns a 500. The model reads the error body as a result, concludes there were no deliveries, and composes a confident reply. Completion rate counts the run as complete, because a reply was produced. Tool selection and argument validity both pass. Only recovery rate sees it: a run containing a failed tool call that never retried, and a step-level assertion that a non-200 must be followed by a retry or an explicit failure to the subscriber.
In production, where the harness is absent, the same failure surfaces through per-tool error rates in performance metric collection and a usage baseline that notices getDeliveries erroring at ten times its normal rate for an hour. The evaluation harness names the defect; the observability baseline catches the next occurrence at three in the morning.
What’s worth remembering
- An agent emits a trajectory, so a single answer-correctness score cannot attribute a failure to a wrong tool, a bad argument, an unretried error, or an unwanted side effect.
- Agent runs are non-deterministic, so the unit that means anything is a pass rate over N runs of the same scripted task, not the result of one pass.
- Gate on task completion rate first: tool selection accuracy and the other route metrics describe nothing when computed over a task the agent never finished.
- Collect the metric set by name (task completion rate, tool selection accuracy, tool-argument validity, steps to completion and turns, recovery rate, cost and latency per completed task, judged reasoning quality) rather than compressing it into one number.
- Amazon Bedrock Model Evaluations scores prompt in and answer out; Amazon Bedrock Agent evaluations scores the invocation including task completion and tool usage, and a judge over AgentCore traces is what reaches reasoning quality in multi-step workflows.
- Any evaluation that calls real tools spends real money and moves real state, so a mocked gateway or a sandboxed tool set is part of the harness design, not a refinement to add later.