The situation
The subscriber help desk runs on an agent hosted in the AgentCore runtime, reaching its tools through a gateway. A message comes in, the agent reasons about it, calls a tool to fetch the account, checks a knowledge base for the refund policy, calls another tool to compute a figure, and writes a reply. Most of the time it is right. This morning it told a subscriber they were owed nothing when they were owed a fortnight’s box, and the only thing anyone can see is that closing sentence.
The final answer is the one artefact that carries none of the information you need. A wrong number at the end could come from a bad tool argument, a stale knowledge-base document, a tool that returned the right value that the model then misread, or reasoning that skipped a step entirely. From the reply alone you cannot tell which, so you cannot fix it, and you cannot tell the subscriber what went wrong with any honesty.
An agent run is a chain of decisions: read the request, pick a tool, pass it arguments, read what comes back, reason to the next step, repeat until done. Debugging one, or proving to an auditor what it did, means reconstructing that chain for a single request. What matters is what to capture, where it lands, and how you pull one run back out of a day’s traffic.
What actually matters
The first thing worth naming is that tracing is a different job from evaluation and from guardrails, and the three are easy to conflate. Tracing answers “why did it do this,” reconstructing one run step by step. Evaluation answers “is it any good,” scoring outputs across many runs against references or a judge. Guardrails answer “stop it doing harm,” blocking or filtering content before it reaches anyone. You need all three, but a guardrail that blocked a response tells you nothing about the reasoning that led there, and an evaluation score tells you the agent is worse this week without telling you which step rotted. Only a trace reconstructs the decision path.
The second is the difference between the model’s reasoning and the plumbing around it. The reasoning layer is the agent’s own thinking: each turn, the tool or knowledge base it chose, the arguments it sent, and what came back. That is the layer that explains why. Underneath, each tool is usually a Lambda calling real systems, and that layer explains what happened when the tool ran: the API it hit, the latency, the error it swallowed. A bad tool call and a bad piece of reasoning look identical from the final answer and completely different once both layers are captured.
The third is verbatim capture versus summary. The trace gives you the shape of the run, but for a real audit or a subtle bug you often need the exact prompt the model saw and the exact completion it produced, byte for byte. Model invocation logging is the feature that records those, the full request and response for each model call, delivered to your own logs and storage. A trace tells you the model called the billing tool; the invocation log tells you the precise text it generated to do so.
The fourth is production reach: getting to one run out of thousands, being alerted when something drifts, and keeping the record long enough to matter. A trace you can only see by re-invoking the agent with tracing switched on in the console is fine for a repro and useless for the incident that already happened. What you want in production is traces, metrics, and logs landing in a store you can query after the fact, with retention you control, tied together by an identifier so one subscriber’s bad morning is a single search rather than a manual hunt.
Underneath all of it: capture is a decision you make before the run, not after. Some of it is account-level setup, some is a toggle on each resource, and some is instrumentation compiled into the agent and its tools. None of it is retroactive, and the run you most want to read is always one that happened before you got around to it. The cheapest time to turn it on is before you need it.
What we’ll filter on
- Can you reconstruct a single run end to end: the reasoning, each tool call, its arguments, and what it returned?
- Are the exact prompts and completions captured verbatim, not just summarised?
- Does the picture join up across the agent and the Lambda tools underneath it, or does it stop at the agent boundary?
- Is it queryable and alertable in production after the fact, or only visible while you re-run the agent?
- How long is it retained, and can it stand up as an audit record of what happened?
The landscape
These are layers that stack, not rivals you choose between. A production setup usually runs several at once, and the first thing to know is how little of it is on by default.
AgentCore’s built-in metrics
Out of the box, AgentCore publishes a set of metrics to CloudWatch for every resource type: the runtime, memory, the gateway, the built-in tools, and identity. Session counts, latency, duration, token usage, and error rates all arrive without you writing anything, and they show up on the CloudWatch generative-AI observability page.
This is the layer that tells you something is wrong. It is aggregate by nature, so it will show the error rate climbing at ten past nine and say nothing about which subscriber, which tool, or which argument. Useful for alarms, useless for the single run you have been asked to explain.
Spans and traces from an instrumented agent
This is the layer that explains why, and it is the one that has to be switched on deliberately. The model has three tiers. A session is the whole conversation with one subscriber. A trace is a single request-response cycle inside it. A span is one unit of work inside that, with a start, an end, a status, and a parent, so a run comes out as a tree: the invocation at the top, reasoning turns, tool calls, and knowledge-base lookups beneath it.
Getting it needs two things that are easy to miss. CloudWatch Transaction Search has to be enabled once for the account, and without it AgentCore cannot deliver spans at all. Then the agent has to be instrumented with the AWS Distro for OpenTelemetry, added to the dependencies and run through opentelemetry-instrument, because AgentCore emits spans by default only for memory resources, and only when tracing is enabled on that memory. Agents and gateways emit metrics for free and spans only when you ask.
If the agent is built on Strands, LangChain, or CrewAI, the framework already speaks OpenTelemetry and the GenAI semantic conventions, so auto-instrumentation carries most of the load. Spans land in the agent’s own log group, /aws/bedrock-agentcore/runtimes/<agent_id>-<endpoint_name>, which is the newer default and keeps spans, logs, and standard output together per agent; older agents deliver to the shared aws/spans group instead.
Model invocation logging
A Bedrock account-level setting, unrelated to AgentCore, that captures the full input and output of every model call and delivers it to CloudWatch Logs, S3, or both. This is the verbatim record: exactly what the model was asked and exactly what it said, byte for byte.
It is the backbone of an audit trail and the thing you reach for when a span’s attributes are not precise enough to explain a subtle failure. It is also a firehose that captures whatever was in the prompt, personal data included, so retention, encryption, and access controls are part of turning it on rather than a later tidy-up.
Distributed tracing into the tool Lambdas
The gateway targets behind an agent are Lambda functions calling downstream systems, and their execution is invisible from the agent’s own spans. Add the AWS Lambda Layer for OpenTelemetry to each one and set AWS_LAMBDA_EXEC_WRAPPER to /opt/otel-instrument, and the function auto-instruments: the services it touched, the latency of each hop, and where an exception was thrown.
One gotcha worth carrying: the ADOT Collector is not supported for agent observability. It is the SDK or the Lambda layer, and reaching for the collector out of habit produces telemetry that never arrives.
Correlation, which is what makes any of it usable
Two identifiers do the stitching. The session id travels on the X-Amzn-Bedrock-AgentCore-Runtime-Session-Id header and is what groups a subscriber’s whole conversation. The trace id travels as X-Amzn-Trace-Id or the W3C traceparent header and is what groups one request across the agent and everything it called.
Propagate both from the front end inward and “show me what happened to subscriber 8c2f at 09:10” is a query. Skip them and you have the same data scattered across log groups with no way to know which rows belong to each other, which is most of the difference between an observability bill and an observability capability.
Not tracing, but next to it
Bedrock model evaluation and RAG evaluation score quality across many runs; Bedrock Guardrails block or filter content in flight. Both produce useful signals and neither reconstructs a decision path. Keep them in the mental map so you do not reach for an evaluation job when what you actually need is one run’s spans.
Evaluation
Side by side
| Signal | Reconstructs one run | Verbatim prompts and completions | Reaches the tool Lambdas | On without setup | Audit-grade retention |
|---|---|---|---|---|---|
| Built-in AgentCore metrics | ✗ (aggregate) | ✗ | ✗ | ✓ | ✓ (as configured) |
| Instrumented spans and traces | ✓ | ✗ (attributes, not full text) | ✓ (with instrumented tools) | ✗ (Transaction Search + ADOT) | ✓ (as configured) |
| Model invocation logging | Partial (per model call) | ✓ | ✗ | ✗ (account setting) | ✓ |
| Distributed tracing (Lambda layer) | Partial (the tool side) | ✗ | ✓ | ✗ (per function) | ✓ (as configured) |
| Evaluation / Guardrails | ✗ | ✗ | ✗ | ✗ | n/a |
Reading it for this incident: the spans tell you which steps the model took and why, invocation logging gives you the exact text it generated, and tool tracing tells you whether the billing Lambda actually returned what the model acted on. No single row does the whole job. The debuggable, auditable setup is spans for the reasoning, the invocation log for the verbatim record, and tool tracing for the plumbing, correlated by session and trace ids. Notice which column is nearly empty: only the aggregate metrics arrive without setup, and aggregate metrics are the one signal that cannot answer the question being asked.
An agent run as a trace
The solution
Spans, for the reasoning. Enable CloudWatch Transaction Search once for the account, add the ADOT SDK to the agent, and every run thereafter is stored as a span tree: a parent for the invocation, children for each reasoning turn, tool call, and knowledge-base lookup, with timing and status on each. This is the signal that answers why, and it is the one you read first when an answer is wrong for no obvious reason. Its limit shapes how you use it: it describes the agent’s view, so it shows that the model called calcRefund with a particular weeks argument and says nothing about what happened inside the Lambda that served the call. Because the spans are stored rather than attached to a live invocation, the run that already failed is still there to read, which is the difference between an incident record and a repro tool.
Model invocation logging, for the verbatim record. Enable it once at the account level and every model call thereafter delivers its full request and response to CloudWatch Logs or S3. When the trace summary says the model “decided no refund was due” and you need to know exactly what it generated to reach that, the invocation log has the literal text. This is the backbone of an audit: it is precise, it is complete, and it lives in storage you control with retention you set. Treat it accordingly. It captures prompts and completions verbatim, which can include personal data, so it needs tight access controls, sensible retention, and encryption, and it is voluminous enough that you plan for its volume rather than discover it on the bill.
Built-in metrics, for health and alarms. AgentCore’s default metrics (sessions, latency, duration, tokens, errors) and your Lambda logs are where you watch the fleet and get told when something moves. You alarm on an error-rate spike or a latency climb and it points you at a window; you then pull the individual traces and invocation logs to see what actually happened in that window. Metrics start investigations; they do not finish them.
Tool tracing, for the plumbing. Add the OpenTelemetry Lambda layer to the functions behind the gateway targets and each tool execution becomes a trace of its own: the downstream services it called, the latency of each, and where an exception was thrown. Correlate those with the agent’s spans and the two failure modes finally separate. “The model passed the wrong argument” shows up in the agent’s spans; “the tool returned a default because the billing service timed out” shows up in the tool’s. Propagating the session and trace ids from the invocation into the tool calls is what lets you stitch one request together across both.
The identifiers, because they are what makes it a system. Send the session id on X-Amzn-Bedrock-AgentCore-Runtime-Session-Id and a trace id as traceparent from the front end, and propagate both inward. That is what turns “show me run 8c2f from this morning” into a query across stored spans rather than a hunt through log groups, and what lets a span in the agent and a segment in a tool Lambda be recognised as the same request. Because the telemetry is OpenTelemetry-shaped, the same identifiers work if you send it somewhere other than CloudWatch, which is a matter of setting DISABLE_ADOT_OBSERVABILITY and pointing the exporter elsewhere.
Worked example
The subscriber wrote in; the agent replied that no refund was due; the subscriber was owed two weeks. The reply is the only thing the help desk agent could see, so the investigation starts by pulling the run.
The span tree for the run lays out the chain. Turn one: fetch the subscription, getSubscription(id=8c2f), returning paused=true. Turn two: consult the refund policy knowledge base, two chunks returned about the refund window. Turn three: compute the refund, calcRefund(id=8c2f, weeks=?). Turn four: compose the reply. The tree shows the shape and already narrows the field: the model did reach the calculation step, so the failure is not a skipped step. The suspect is the weeks argument it passed.
The invocation log for that model call has the verbatim completion, and there it is: the model generated weeks=0, having read the pause as covering the current week only, when the subscriber had been paused for two delivery cycles. The exact text of the reasoning shows the misread. The tool did nothing wrong, it computed a zero-week refund as zero.
To be sure the tool was innocent, the tool trace for calcRefund, correlated by trace id, confirms it: the Lambda received weeks=0, called the billing service, which responded in 40ms with AUD$0.00, no error, no timeout. The plumbing was healthy. The defect was upstream, in how the model turned the pause history into an argument.
That is a fix you can now make with confidence: the pause data the model reads is ambiguous about multi-cycle pauses, so you tighten what the account tool returns and add an instruction about counting cycles. Without the trace you would have been guessing between a tool bug, a stale policy document, and a reasoning error. With it, the wrong step is named, the verbatim proof is on file, and the tool is cleared. If an auditor asks later what the agent did for subscriber 8c2f on this date, the same three artefacts answer them.
What’s worth remembering
- The final answer is the one artefact that hides the reason; debugging or auditing an agent means reconstructing the run behind it, not reading the reply.
- Stored spans are the layer that explains the model’s reasoning, and on AgentCore they cost setup: Transaction Search for the account, ADOT in the agent, and tracing enabled per resource.
- Model invocation logging is the verbatim record, the exact prompts and completions, delivered to your own logs or S3, and it is the backbone of an audit trail.
- The agent’s spans stop at the agent; tracing the tool Lambdas shows what they actually did, and correlating the two by session and trace id separates a bad argument from a bad tool.
- Only the aggregate metrics arrive for free, and they are the one signal that cannot explain a single run; everything that can is a before-the-run decision, and none of it is retroactive.
For a run whose whole shape you need to defend, from the model’s reasoning down into the tool it triggered, no single signal is enough. The spans explain the decisions, the invocation log proves them word for word, and the tool trace shows whether the plumbing was at fault, all stitched together by the session and trace ids. That combination, closer to the way a supervisor over collaborators multiplies the number of runs you will one day have to explain, is what turns a wrong answer from a mystery into a fix.