Exam-style
An agent with six tools has been giving noticeably worse answers all week. The error rate on the agent itself is flat, latency looks normal, and nobody has deployed a prompt change. The team needs to know whether the model has started choosing a different tool for the same kind of request, or whether one of the downstream tools has started returning empty defaults instead of real data. What gives them that?
Reveal the answer
D. Emit a metric per tool from the agent's spans (call volume, share of selections, duration, error rate), compare each against the previous week's baseline, and correlate the agent's spans with traces from the tool Lambdas on a shared session and trace id
Two different faults are producing one symptom, and only an aggregate view separates them. Per-tool metrics derived from the agent’s spans give call pattern tracking: how often each tool is selected, how the selections split across the catalogue, and the duration and error rate of each one on its own. Hold those against usage baselines for anomaly detection built from the previous week and the shift names itself. A move in the selection distribution is the model behaving differently. A move in one tool’s error rate or duration is plumbing. Carry the same session and trace ids from the agent’s spans into the tool Lambdas’ own traces. That is what tool calling observability and multi-agent coordination tracking looks like in practice. It lets the model’s choice and the tool’s execution be read apart rather than as one number. Of the rest: model invocation logging keeps a verbatim record of prompts and completions, enough to settle one complaint, with nothing in aggregate and nothing from inside a tool; the runtime’s default session and latency metrics report fleet health with no per-tool breakdown, which is why the agent’s own error rate stayed flat while the answers rotted; CloudTrail records control-plane API calls, not the tool executions inside a run; and debug logging is a bespoke pipeline that answers this once, at a log volume the team pays for forever.
Q. Answers are degrading, the agent’s error rate is flat, and six tools sit behind it. Did the model start picking differently, or did a tool start lying?
A. Per-tool metrics off the agent’s spans, compared against last week. Call volume and share of selections per tool, plus duration and error rate per tool. Propagate the trace and session ids the run already carries into the tool Lambdas, and a slow or failing tool lands on the same timeline as the decision that called it.
Why? The selection distribution and the per-tool error rate fail in opposite directions, and the agent’s own error rate hides both. A tool that returns an empty default is a success as far as the agent is concerned, so it never shows up as an error. The model quietly picking tool four where it used to pick tool two never shows up at all. Baseline both, alarm on the deviation, and the same instrumentation carries over when one agent starts calling another and the tool at the far end is a whole second agent.