Exam-style
An agent with six tools has given noticeably worse answers all week. Its own error rate is flat, latency is normal, and nobody changed the prompt. The team needs to know whether the model is now selecting a different tool for the same request, or whether one tool is returning empty defaults instead of real data. What gives them that?
Reveal the answer
D. Emit per-tool metrics from the agent's spans (call volume, share of selections, duration, error rate), compare each against last week's baseline, and correlate them with the tool Lambdas' traces on a shared session and trace id
Two faults produce one symptom, and only an aggregate view separates them. Per-tool metrics from the agent’s spans give call pattern tracking: how often each tool is selected, how selections split across the catalogue, and each tool’s own duration and error rate. Hold those against usage baselines for anomaly detection built from the previous week. A move in the selection distribution points at the model; a move in one tool’s duration or error rate points at the plumbing. Carrying the same session and trace ids into the tool Lambdas’ traces is tool calling observability and multi-agent coordination tracking in practice. Of the rest: model invocation logging records the request and response bodies of Converse and InvokeModel calls, with nothing in aggregate and nothing from inside a tool; the AgentCore runtime publishes session count, latency and error metrics by default with no per-tool breakdown, which is why the agent’s own error rate stayed flat; CloudTrail gateway data events record each tools/call with the tool name but redact the arguments and the result, so a diff shows a selection shift and never an empty response; and debug logging answers this once and is charged for as long as it runs.
Q. Answers are degrading, the agent’s error rate is flat, and six tools sit behind it. Did the model start selecting differently, or did a tool start returning empty results?
A. Per-tool metrics off the agent’s spans, compared against last week. Call volume and share of selections per tool, plus duration and error rate per tool. Propagate the trace and session ids the run already carries into the tool Lambdas, and a slow or failing tool lands on the same timeline as the decision that called it.
Why? The selection distribution and the per-tool error rate fail in opposite directions, and the agent’s own error rate covers neither. A tool that returns an empty default still returns successfully, so it never registers as an error. A shift from tool two to tool four never registers either. Where the tools are AgentCore Gateway targets, the gateway already publishes invocation, duration and error metrics with the tool name as a dimension, which covers half of that without writing any instrumentation. Baseline both, alarm on the deviation, and the same approach carries over when one agent starts calling another and the tool at the far end is a second agent.