Exam-style
A refunds agent scores 92% on final-answer correctness against the golden set, and the score has been stable for a month. Operations report that a handful of subscribers each week receive the same refund twice, and in every case the agent's closing summary describes the refund accurately, in the singular. What should the team measure to catch this?
Reveal the answer
C. Task completion rate, plus tool selection accuracy and tool-argument validity read off the recorded trajectory
Answer correctness scores the last thing the agent says. Calling the refund tool twice and then writing one accurate sentence about it passes that check every time, so a stricter judge and a bigger sample both sharpen a measurement that cannot reach the failure. Task completion rate measures something else: whether the run finished in the state it was meant to reach, one refund issued. Tool selection accuracy and tool-argument validity, read off the recorded trajectory, then show the duplicate call and the arguments it carried. Amazon Bedrock AgentCore Evaluations scores agent traces at that level; Bedrock evaluations cover models and RAG sources, not trajectories. Model invocation logging records request and response bodies for hand sampling and produces no score to track. A guardrail filters text in prompts and responses, not tool calls.
Q. The refunds agent scores 92% on final-answer correctness and refunds are still going out twice. What do we measure?
A. Task completion rate, and the route the run took: tool selection accuracy and tool-argument validity over the recorded trajectory. Amazon Bedrock AgentCore Evaluations scores agent traces at session, trace and tool-call level, and its trajectory evaluators compare the actual sequence of tool calls against the expected one. A second call to the payments tool fails that comparison.
Why? An answer-only metric scores the reply, so the route never enters the number. The route is where a side-effecting agent moves money. Raising the judge’s strictness or the golden-set sample size only makes an unrelated number more precise.