Monitoring a Production Bedrock App

The situation

A product team has shipped a customer-facing assistant on Amazon Bedrock. One Claude model sits behind three features: a chat panel that streams answers, an overnight document-summarisation batch, and an inline “explain this” helper. Traffic has grown from a demo to real load, and three complaints landed in the same week.

Finance says the Bedrock line on the bill more than doubled month on month. Nobody can say which feature is responsible, or whether one of them is looping. Support has forwarded screenshots where the streaming chat sat blank for eight or nine seconds before any text appeared, and users assumed it had hung. A spot-check also turned up two summaries containing figures that were not in the source document.

The team has CloudWatch switched on and can see that Bedrock is being called a lot. They cannot tell cost, latency and quality apart, attribute any of them to a feature, or catch the next regression before a customer does.

What actually matters

These are three problems with three different homes. Cost, latency and throttling are operational numbers the platform publishes about itself. Quality is a property of the content. A dashboard built around “is Bedrock healthy” will cover the first two and skip the third, which is the one that produced the invented figures.

Cost is driven by tokens, not requests, so a request count tells you very little about spend. Two calls with the same invocation count can differ tenfold, because one put a whole document in the context and the other asked a one-line question. The useful signal is input and output token counts, and the useful question is per-feature. Attribution beats the aggregate: you cannot fix a bill you cannot break down.

Latency has a shape that an average hides, and streaming sharpens that. For a streaming feature the number a user feels is the wait before any text appears, which is a different quantity from total generation time. A reply that streams for six seconds but starts inside one feels fast. A reply that starts after eight feels broken, even if it finishes sooner. Bedrock publishes both numbers, separately, and collapsing them into one is the mistake.

Throughput and throttling are the capacity story. Bedrock applies per-model requests-per-minute and tokens-per-minute quotas in each Region. Traffic above them comes back as throttling errors rather than as a slowdown. Throttled requests count as neither invocations nor errors, so a latency graph will not show them at all. When throttles climb, the fix is a capacity change rather than a code change.

Quality is the part that needs building, though less of it than the team assumes. Bedrock can report that an invocation succeeded, returned 400 tokens and took 900 milliseconds, while those 400 tokens contain a fabricated figure. No runtime metric scores correctness. Two mechanisms narrow the gap. A guardrail can compare a response against a source document at request time and filter it when the response is not grounded in that source. An offline scoring pass over captured prompts and completions turns the spot-check into a trend. Both have to be set up, and neither arrives with the metrics.

What we’ll filter on

1. Does the platform publish the signal, or do you have to construct it? Tokens, latency, time-to-first-token and throttles are published; a correctness score is not.
2. Can it be attributed to one feature, rather than only to the model?
3. Does it match what a user experiences, rather than a server-side mean?
4. Can you alarm on it, so a regression pages someone?
5. What does it cost to run, in storage, tokens and review time?

The landscape

Runtime metrics. The bedrock-runtime endpoint publishes to the AWS/Bedrock namespace, dimensioned by ModelId. Invocations counts successful calls. InvocationLatency measures from the request being sent to the last token arriving. TimeToFirstToken measures to the first token, and is published for the two streaming operations, ConverseStream and InvokeModelWithResponseStream. InputTokenCount and OutputTokenCount carry the token volumes, with CacheReadInputTokenCount and CacheWriteInputTokenCount splitting out prompt caching. InvocationClientErrors, InvocationServerErrors and InvocationThrottles cover failure. EstimatedTPMQuotaUsage approximates quota consumption, and the documentation warns against treating it as the sole input to capacity planning. These are ordinary CloudWatch metrics: graph them, take percentiles, alarm on them.

Alarms, dashboards and anomaly detection. On top of those metrics sits the operational layer. Alarm on rising InvocationThrottles, on p99 InvocationLatency, on p99 TimeToFirstToken, and on an OutputTokenCount sum stepping outside its normal band. A static threshold on a token metric goes stale within a month on growing traffic. CloudWatch anomaly detection learns the expected band from the metric’s own history, daily and weekly shape included, and alarms when the metric leaves it. That catches retry storms, loops and slow prompt-size creep alike. AWS Cost Anomaly Detection is the billing-side equivalent, watching the Bedrock spend curve rather than the token counts.

Tracing the call path. Metrics say p99 got worse; they do not say which hop got worse. AWS X-Ray times the retrieval query, the Bedrock call and the application work either side as separate segments, and Application Signals maps the services around them. Annotate segments with the model id and the prompt version and one slow response can be read hop by hop. Enabling CloudWatch Transaction Search ingests spans as structured logs so individual traces stay searchable without span-level sampling.

Model invocation logging. This is off by default, and it is configured per account per Region. Once enabled it captures the full request body, response body and metadata for Converse, ConverseStream, InvokeModel and InvokeModelWithResponseStream, and delivers them to Amazon S3, to CloudWatch Logs, or to both. Bodies up to 100 KB appear inline in the record; larger bodies and binary data are written as separate S3 objects under the data prefix. Each record carries the request id, the model or inference profile id, the caller’s identity.arn, and the input and output token counts. Sending the logs to CloudWatch Logs also switches on the pre-built model invocation dashboards in CloudWatch generative AI observability, where a request id opens its own input and output. The record is what you need for debugging one bad answer, for audit, and for offline scoring, because you cannot score outputs you never kept. It is also the sensitive one: full prompts and completions can contain customer data, so the destination needs the access controls and retention you would apply to any other store of user content.

Attribution. There are two mechanisms and they answer different questions. An application inference profile is a resource that references one model and carries cost allocation tags. You call it by putting the profile ARN in the modelId field, and after you activate the tags in the Billing console they flow to Cost Explorer and to Cost and Usage Reports. The grain there is per usage type per day, not per request, and the tags are not retroactive. Per-request metadata is the other half: up to 16 key-value pairs per call, set as requestMetadata on the Converse APIs or the X-Amzn-Bedrock-Request-Metadata header on InvokeModel, recorded in the invocation log and nowhere else. It gives per-prompt token detail, and it never reaches Cost Explorer. Profiles report under their own id in the ModelId metric dimension, so per-feature CloudWatch metrics come with them. Note that application inference profiles are rejected by the Responses and Chat Completions APIs.

Guardrails and contextual grounding. Guardrails publish to the AWS/Bedrock/Guardrails namespace, where InvocationsIntervened counts the requests a guardrail acted on, split by GuardrailContentSource for input against output and by GuardrailPolicyType for which policy fired. The contextual grounding check is the policy that matters here. Given a grounding source, the user query and the model response, it produces grounding and relevance confidence scores, and filters the response when either falls below a threshold you set between 0 and 0.99. It runs on output only, and the documented limits are 100,000 characters of grounding source, 1,000 of query and 5,000 of response. Conversational chatbot use is outside what it supports; summarisation and question answering are inside it. So the summariser can have a runtime check, and the ContextualGroundingPolicy slice of InvocationsIntervened becomes a trend line for how often the model produced ungrounded text.

The scoring pass. A guardrail catches ungrounded output one response at a time. It does not tell you whether last Tuesday’s prompt change made the feature worse. For that, sample the logged completions on a cadence and score the sample. Amazon Bedrock evaluations runs the managed version: a judge-model evaluation job scores responses with a second model and explains each score, and a RAG evaluation job computes correctness, completeness, helpfulness, citation precision and citation coverage. Both run against a dataset you supply, so the workflow is to export a sample of the invocation logs and feed it in. Human review of a small sample and user feedback in the app (thumbs up and down, edit-and-resend, abandonment) are the cheaper proxies alongside it. Tag every score with the prompt version. A pass rate that slides while the prompt text stays unchanged points at the model or the corpus moving underneath you.

Business metrics. Cost, latency and quality all measure the machine. Containment rate, escalation rate, edit-and-resend rate, abandonment and task completion measure what the feature did for people. None of them come from Bedrock. They come from the application’s own events, joined to the invocation logs by request id, which makes the capture application work. Publish them as custom metrics next to the operational ones and finance’s question becomes answerable as cost per resolved session rather than cost per invocation.

Evaluation

Side by side

Signal	Source	Published or built	Per-feature	Alarmable	Catches the invented figure
Invocations	AWS/Bedrock	Published	Via inference profile	✓	✗
Token counts	InputTokenCount / OutputTokenCount	Published	Via inference profile	✓	✗
Total latency	InvocationLatency	Published	Via inference profile	✓	✗
Time-to-first-token	TimeToFirstToken (streaming ops)	Published	Via inference profile	✓	✗
Throttling	InvocationThrottles	Published	Via inference profile	✓	✗
Prompts and completions	Model invocation logging	Published, off by default	Yes, in the record	✗ (a record)	Only once scored
Grounding check	Guardrails InvocationsIntervened	Published	By guardrail and policy	✓	✓ at request time
Correctness scores	Bedrock evaluations, human review	Built on the logs	Yes, by design	✓ (derived score)	✓ as a trend
Token anomaly band	Anomaly detection on the token metrics	Derived band	Via inference profile	✓	✗
Call-path traces	X-Ray segments and annotations	Published once instrumented	By feature and prompt version	✗ (a trace)	✗
Business metrics	App events joined by request id	Built	Yes, by design	✓ (custom metric)	Indirectly

The solution

Cost. Break the aggregate down before anything else, because the finance complaint has no other answer. Give each feature its own application inference profile, tagged, and point the chat panel, the summariser and the helper at their own. Spend then splits by feature in Cost Explorer, and the CloudWatch token metrics report under each profile’s id. Watch the token counts rather than the invocation count. A summariser that has crept from 2,000-token to 8,000-token inputs shows up in InputTokenCount weeks before the bill lands, and a chat feature producing 3,000-token rambles shows up in OutputTokenCount. Let anomaly detection draw the band instead of picking a threshold, because a number chosen for this month’s traffic is wrong by next month. Keep AWS Cost Anomaly Detection on the Bedrock spend line as the backstop.

Latency. Alarm on p99 TimeToFirstToken for the two interactive features and on p99 InvocationLatency for the overnight batch, where nobody is waiting on a first token. Track percentiles rather than means, since a healthy p50 hides the p99 that generated the support screenshots. If total latency rises while output tokens per second holds steady, the responses have got longer rather than the service slower, and the output token counts will confirm it. When a percentile does move, the X-Ray traces say whether the extra seconds went on retrieval, on the model, or on your own code. A CloudWatch Synthetics canary adds coverage when traffic is quiet: a scripted probe on a schedule, down to once a minute, publishing metrics under CloudWatchSynthetics that you can alarm on at three in the morning.

Throughput and throttling. Put InvocationThrottles on a dashboard and an alarm from the first day, because throttles are lost requests and a latency graph never shows them. If they climb under normal load, the options are all capacity ones. Request a quota increase. Move the overnight summariser to batch inference, which runs asynchronously against files in S3 at half the on-demand token rate with a 24-hour completion window, and stops it competing with the interactive traffic. Or reserve input and output tokens-per-minute on the Reserved tier for the steady interactive baseline, on a one or three month commitment, with traffic above the reservation overflowing to Standard. One caveat on the batch move: batch inference supports neither tool calling nor structured output, so a summariser that uses either stays synchronous.

Quality. Enable model invocation logging first. Without the captured completions there is nothing to inspect after the fact, and every incident stays a screenshot and a shrug. Send the logs to S3 with the access controls and retention that customer content requires, and a filtered slice to CloudWatch Logs for searching. Then attach a guardrail with the contextual grounding check to the summariser, passing the source document as the grounding source and the user question as the query, so an ungrounded summary is filtered before a customer reads it. Alarm on the ContextualGroundingPolicy slice of InvocationsIntervened, which climbs when the model starts producing more ungrounded text. Behind that, run the sampled scoring pass: export sampled completions and score them with a Bedrock judge-model or RAG evaluation job. Scoring a sample against its sources covers the mechanics. Publish the pass rate as a custom metric, tagged with the prompt version, so a change that gained two points of accuracy while tripling the context is visible the day it ships.

Publishing the built signals. The scores and business numbers all get described as “a custom metric”, which skips how they arrive. The cheap default is the CloudWatch embedded metric format. The service that already writes a structured log line for the interaction embeds the metric values in that same JSON, and CloudWatch extracts them on ingest. One write gives you the metric to alarm on and the record to query, with the request id in both, so a spike on the graph leads straight to the interactions behind it. Keep PutMetricData for the cases with no log line to ride on, such as an offline job publishing a batch pass rate.

Business impact. Wire the application’s own events into the same dashboard once the operational three are in place: containment, escalation, edit-and-resend, abandonment and task completion, keyed by request id so they join the invocation logs and the traces. A summariser that costs twice as much and halves the reading a person has to do is a good trade, and cost per invocation is the one view that cannot show it.

Worked example

The finance complaint closes fastest, and it shows the published metrics and the built attribution working together. The team creates three application inference profiles over the same model, one per feature, each tagged with the feature name. The chat panel, the summariser and the helper each call their own profile ARN. Nothing about the model or the prompts changes; only the call path is labelled.

Within a day the token metrics tell the story the aggregate could not. Chat and the inline helper sit flat. The summariser’s InputTokenCount has roughly quadrupled over the month, tracking a change that let it pass whole documents into context instead of a trimmed extract. It never looped and never errored, so no failure metric moved. It was feeding the model four times the tokens per run, and on a per-token bill that accounts for the doubling.

The fix is now scoped to one feature: trim or pre-summarise the document, or move the batch onto provisioned throughputProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. so its cost is a fixed line rather than a per-token one. The alarm on InputTokenCount for that profile means the next such creep pages the team rather than appearing on an invoice five weeks later. The same tags make the quality sampling per-feature, so the summariser that caused the cost scare is also the one whose grounding scores get watched closest.

What’s worth remembering

1. Cost, latency and quality live in three different places, and one “is Bedrock up” dashboard covers the easy two and misses the one behind the worst complaints.
2. Cost tracks tokens, not requests, so watch InputTokenCount and OutputTokenCount; an invocation count says almost nothing about spend.
3. Bedrock publishes TimeToFirstToken for ConverseStream and InvokeModelWithResponseStream, alongside InvocationLatency for the full response; alarm on both as percentiles.
4. Model invocation logging captures prompts, completions and metadata to S3 and CloudWatch Logs, with bodies over 100 KB written to S3 separately, and it is off until you enable it per Region.
5. Tagged application inference profiles carry cost to Cost Explorer at a daily grain and give per-feature CloudWatch metrics; per-request metadata gives per-prompt detail in the logs and never reaches the bill.
6. The Guardrails contextual grounding check filters ungrounded responses at request time, and a sampled scoring pass over the logs turns correctness into a trend you can alarm on.