Exam-style
A refunds agent scores 92% on final-answer correctness against the golden set, and the score has been stable for a month. Operations report that a handful of subscribers each week receive the same refund twice, and in every case the agent's closing summary describes the refund accurately, in the singular. What should the team measure to catch this?
Reveal the answer
C. Task completion rate, plus tool selection accuracy and tool-argument validity read off the recorded trajectory
Answer correctness scores the last thing the agent says. Calling the refund tool twice and then writing one accurate sentence about it passes that check every time, so a stricter judge and a bigger sample both sharpen a measurement that cannot see the failure at all. Task completion rate asks a different thing: did the run finish in the state it was meant to reach, one refund issued. Tool usage effectiveness metrics over the recorded trajectory, tool selection accuracy and tool-argument validity, then show the duplicate call and the arguments it carried. Invocation logging keeps the raw prompts and completions but is not a metric, and a guardrail constrains text rather than side effects.
Q. The refunds agent scores 92% on final-answer correctness and refunds are still going out twice. What do we measure?
A. Task completion rate, and tool usage effectiveness over the recorded trajectory: tool selection accuracy and tool-argument validity, plus reasoning quality assessment across the multi-step run. Amazon Bedrock Agent evaluations score the trajectory as well as the output.
Why? An answer-only metric is blind to the route the agent took. The route is where a side-effecting agent does its damage. Raising the judge’s strictness or the golden-set sample size only makes an unrelated number more precise.