Monitoring a Production Bedrock App

The situation

A product team has shipped a customer-facing assistant on Amazon Bedrock. It runs a single Claude model behind three features: a chat panel that streams answers, a document-summarisation job that runs in batches overnight, and an inline “explain this” helper embedded in the app. Traffic has grown from a demo to real load, and three separate complaints have landed in the same week.

Finance has noticed the Bedrock line on the AWS bill more than doubled month on month, and nobody can say which of the three features is responsible or whether one of them is looping. Support has forwarded a handful of screenshots where the streaming chat sat blank for eight or nine seconds before any text appeared, and users assumed it had hung. And a manual spot-check turned up two summaries that invented figures that were not in the source document, which is the kind of thing that erodes trust faster than a slow response ever will.

The team has CloudWatch switched on and can see that “Bedrock is being called a lot”. What they do not have is a way to tell cost, latency, and quality apart, attribute any of them to a feature, or catch the next regression before a customer does. The underlying question is which signals Bedrock gives you, which you have to construct, and where each one lives.

What actually matters

The first thing worth naming is that these are three different problems with three different homes, and conflating them is why the team is stuck. Cost and latency and throttling are operational metrics the platform emits about itself. Quality is a property of the content, and the platform has no opinion about content. Any monitoring design that treats “is Bedrock healthy” as one dashboard will measure the two easy things and silently skip the one that actually generated the complaint about invented figures.

Cost on Bedrock is driven by tokens, not requests, so a request count tells you almost nothing about spend. Two calls with the same invocation count can differ tenfold in cost because one stuffed a whole document into the context and the other asked a one-line question. That means the useful cost signal is input and output token counts, and the useful cost question is per-feature: which of the three features is burning the budget, and is any of them regressing toward longer prompts or runaway output. Attribution matters more than the aggregate, because you cannot fix a bill you cannot break down.

Latency has a shape that a single average hides, and streaming makes this sharper. For a streaming feature the number the user actually feels is time-to-first-token, the wait before anything appears, which is a different quantity from total generation time. A response that streams for six seconds but starts in under one feels fast; a response that starts after eight seconds feels broken even if it finishes sooner overall. Server-side invocation latency and client-perceived time-to-first-token are both worth having, and they are not the same measurement.

Throughput and throttling are the capacity story. Bedrock enforces account-level and model-level quotas, and on-demand traffic that pushes past them comes back as throttling errors rather than a slowdown. If throttles are climbing you are leaving requests on the floor, and the fix is a capacity decision (request a quota increase, or move the steady load onto provisioned throughputProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not.), not a code change. This signal has to be visible before customers feel it as failures.

Quality is the one the platform cannot see. Bedrock will report that an invocation succeeded, returned 400 tokens, and took 900 milliseconds, while those 400 tokens contain a fabricated number. There is no server-side metric for correctness, faithfulnessFaithfulnessWhether every claim in an answer is actually supported by the source it was given, regardless of whether it happens to be true., or tone, because those are judgements about meaning. So quality monitoring is something you build on top: capture the actual prompts and completions, sample them, and score the sample by some method that understands content, whether that is a second model acting as a judge, a human reviewer, or a proxy signal like how often a guardrail had to intervene or how often users reacted badly. The distinction that runs through the whole design is free metrics versus built signal: the platform gives you the operational numbers, and you construct the quality one or you do not have it.

What we’ll filter on

1. Is the signal emitted by the platform, or does it have to be constructed? Cost, latency, and throttling are emitted; quality is constructed.
2. Can it be attributed to a specific feature? Per-feature cost and error breakdown is worth more than a single aggregate.
3. Does it capture the user-perceived shape, not just a server-side average? Time-to-first-token for streaming, percentiles rather than means.
4. Does it support alarms, so a regression pages someone instead of waiting for a complaint?
5. What does it cost to run, in storage and in review effort? Full-payload logging and human scoring are not free.

The landscape

CloudWatch metrics from Bedrock. Bedrock publishes runtime metrics to the AWS/Bedrock namespace automatically, at no extra cost, dimensioned by model. The ones that carry the load here are Invocations (call count), InvocationLatency (server-side processing time), InputTokenCount and OutputTokenCount (the token volumes that drive both cost and latency), and the error and throttle counters InvocationClientErrors, InvocationServerErrors, and InvocationThrottles. These are standard CloudWatch metrics, so you can graph them, take percentiles rather than averages, and set alarms on them. They tell you how much, how fast, and how often it failed. They tell you nothing about what was in the request or how good the answer was.

CloudWatch alarms and dashboards. On top of those metrics you build the operational layer: an alarm on rising InvocationThrottles so capacity pressure pages before it becomes user-visible failures, an alarm on p99 InvocationLatency, an alarm on an OutputTokenCount sum that jumps past its normal band (the cheapest early warning for a feature that has started looping or ballooning its output). A dashboard puts token counts, latency percentiles, and error rates side by side so the three failure directions are legible at a glance. This is all standard CloudWatch, nothing Bedrock-specific beyond the metric names. A CloudWatch Synthetics canary belongs here too: one golden prompt fired at the live endpoint every few minutes turns availability and first-token latency into a signal that exists even when real traffic is quiet, which matters for a feature whose overnight hours are empty.

CloudWatch anomaly detection. A static threshold on a token metric is a guess, and traffic that grows week on week makes it a guess that goes stale within a month. CloudWatch anomaly detection learns the normal band for InputTokenCount and OutputTokenCount from their own history, daily and weekly shape included, and alarms when the metric steps outside it. That catches token burst patterns (a retry storm, a loop, a tool call that has started calling itself) and the slower prompt-size creep that no fixed number would have been set low enough to see. A metric whose normal band moves with traffic wants an alarm that moves with it, which no fixed number ever does. AWS Cost Anomaly Detection is the billing-side equivalent, watching the Bedrock spend curve rather than the token counts and alerting when the shape of the spend changes, so a regression that arrives as money gets noticed even if nobody put an alarm on the metric that caused it.

Traces across the call path. Metrics tell you that p99 got worse; they do not tell you which hop got worse. AWS X-Ray gives you performance tracing across the whole request, timing the retrieval query, the Bedrock call, any tool invocations, and the application work either side of them as separate segments. Annotate those segments with the model id, the prompt version, and the token counts and it becomes FM interaction tracing, so one slow or one bad response can be opened up and read hop by hop instead of inferred from a graph. The same traces support API call profiling for prompt-completion patterns: group the Bedrock segments by prompt version and the shapes fall out, which versions produce long completions, which ones retry, and which ones sit at the top of the latency distribution.

Bedrock model invocation logging. This is the piece you have to turn on deliberately; it is off by default. Once enabled in the Bedrock settings for the account and region, it captures the full request and response for invocations (the complete prompts and completions) along with metadata, and delivers them to Amazon S3, to Amazon CloudWatch Logs, or to both. Large payloads and any image or embedding data are written to an S3 bucket you nominate. This is the record you need for debugging a specific bad answer, for audit, and above all for offline quality analysis, because you cannot score outputs you never kept. It is also the sensitive one: full prompts and completions can contain customer data, so the log destination needs the same access controls and retention thinking as any other store of user content.

Application inference profiles for cost attribution. An application inference profile wraps a foundation model with your own tags, and calls made through the profile carry those tags into cost and usage tracking. That is the clean way to answer “which feature spent the money”: give the chat panel, the summariser, and the inline helper each their own tagged profile, and the token spend splits by feature in Cost Explorer and in per-profile CloudWatch metrics instead of collapsing into one undifferentiated model line. Without something like this, on-demand spend is attributed to the model, not to the feature that made the call.

Bedrock Guardrails signals. If the app runs Bedrock Guardrails, the rate at which the guardrail intervenes (blocking or masking content, on input or output) is a genuine quality-adjacent signal that the platform does surface. A climbing intervention rate says either the inputs are getting more adversarial or the model is drifting toward output the policy rejects, and either way it is worth an alarm. It is a proxy for quality, not a measure of correctness, but it is one you get without building a scorer.

The quality layer you build. Nothing above measures whether the summary was faithful to the document. That layer is yours to construct on top of invocation logging: sample the captured completions on some cadence, and score the sample. The score can come from an LLM-as-a-judge pass (a second model prompted to rate faithfulness or correctness against the source), from human review of a small sample, or from user feedback signals wired into the app (thumbs up and down, edit-and-resend, abandonment). Whichever you pick, it runs offline against the logs, it works on a sample rather than every call because scoring is not free, and it is the only thing in the whole design that actually looks at meaning. Give the built score names that survive a conversation with someone outside the team. Hallucination rate is the share of sampled completions that assert something the source does not support, and it is the number that answers the invented-figures complaint directly. Prompt effectiveness is a prompt version’s pass rate set against the tokens it burns to reach it, which makes a prompt that got more accurate by tripling its context look like what it is. Track both by prompt version and response drift shows up on its own, because a pass rate that slides while the prompt sits unchanged is the model or the corpus moving underneath you.

The business layer. Cost, latency and quality all measure the machine. The layer above them measures what the feature did for the people using it, and it is what a dashboard needs before anyone outside the team will read it. The metrics are containment rate (the share of sessions resolved without a human picking them up), escalation rate, edit-and-resend rate, abandonment, and task completion. None of them come from Bedrock. They come from the application’s own event store, joined to the invocation logs by interaction id, which means the capture is application work: an event when a session closes, an event when a human takes it over, an event when the user edits the answer before sending it on. Publish them as custom metrics in the same namespace as the operational ones and business impact metrics with custom dashboards stop being a separate reporting exercise; the token counts and the containment rate sit on one screen, and the question finance actually asked becomes answerable as cost per resolved session rather than cost per invocation. A feature whose spend doubled while its containment rate doubled with it is a different story from one whose spend doubled on its own.

Evaluation

Side by side

Signal	Source	Emitted or built	Per-feature attribution	Alarmable	Catches the invented figure
Invocation count	CloudWatch Invocations	Emitted (free)	Via tagged inference profile	✓	✗
Token counts (cost)	CloudWatch InputTokenCount / OutputTokenCount	Emitted (free)	Via tagged inference profile	✓	✗
Server latency	CloudWatch InvocationLatency	Emitted (free)	By model dimension	✓	✗
Time-to-first-token	Client instrumentation on the stream	Built	By feature (you own it)	✓ (custom metric)	✗
Throttling	CloudWatch InvocationThrottles	Emitted (free)	By model dimension	✓	✗
Full prompts and completions	Model invocation logging to S3 / CloudWatch Logs	Emitted, but off by default	Yes, in the log record	✗ (it is a record, not a metric)	Only if you read or score it
Guardrail intervention rate	Bedrock Guardrails	Emitted (free)	By guardrail	✓	Partly (policy hits only)
Faithfulness / correctness	LLM-as-a-judge, human review, user feedback	Built on top of logs	Yes, by design	✓ (on the derived score)	✓
Token anomaly band	CloudWatch anomaly detection on the token metrics	Emitted metric, derived band	Via tagged inference profile	✓	✗
End-to-end traces	AWS X-Ray segments and annotations	Emitted once instrumented	By feature and prompt version	✗ (a trace, not a metric)	✗ (shows the path, not the content)
Golden-prompt canary	CloudWatch Synthetics	Built	By endpoint	✓	Partly (one prompt only)
Business impact metrics	App event store joined to invocation logs by interaction id	Built	Yes, by design	✓ (custom metric)	Indirectly (escalations and edits climb)

The solution

Cost. Turn the aggregate into a per-feature breakdown before doing anything else, because the finance complaint cannot be answered otherwise. Give each of the three features its own application inference profile, tagged, so spend splits by feature in Cost Explorer and the token-count metrics carry the tag. Then watch the token counts, not the invocation count, since tokens are what the bill is made of. A summariser that has crept from 2,000-token to 8,000-token inputs shows up as a rising InputTokenCount long before the monthly bill lands, and a chat feature that has started producing 3,000-token rambles shows up in OutputTokenCount. Alarm on the sums stepping outside their normal band, and let CloudWatch anomaly detection draw that band rather than picking a number, because a threshold set for this month’s traffic is wrong by the next one. That single alarm is the earliest, cheapest catch for both a runaway loop and a quiet prompt-size regression, and AWS Cost Anomaly Detection on the Bedrock spend line sits behind it as the backstop for anything that only becomes visible as money.

Latency. Measure it as the user feels it, which for the streaming chat means time-to-first-token, and that is not something Bedrock hands you as a metric. When you call the streaming API, timestamp the request and timestamp the first content chunk off the stream, and publish the difference as a custom CloudWatch metric. Keep the server-side InvocationLatency too for the batch summariser, where total time is what matters and there is no user waiting on a first token. Track both as percentiles, because a p50 that looks fine can hide a p99 that is generating the support screenshots. Alarm on p99 time-to-first-token for the interactive features and on total latency for the batch one. Add a CloudWatch Synthetics canary firing one golden prompt at the live endpoint every few minutes, so availability and first-token latency are still being measured at three in the morning when no customer is there to measure them for you. When a percentile does move, the X-Ray traces are what tell you whether the extra seconds went on retrieval, on the model, or on your own code.

Throughput and throttling. Put InvocationThrottles on a dashboard and an alarm from day one, because throttles are lost requests, not slow ones, and they are invisible in a latency graph. If they climb under normal load you have a capacity decision to make: request a service quota increase for the model, move the overnight batch summariser onto the Batch Inference API so it stops competing with the interactive features for the synchronous on-demand pool, or reserve tokens-per-minute on the Reserved tier for the steady interactive baseline, with traffic above the reservation overflowing to Standard. This is a knob-turning fix, and the alarm is what tells you to reach for the knob.

Quality. This is the one the team does not have at all, and it is the reason a fabricated figure reached a customer. Start by enabling model invocation logging, because without the captured completions there is nothing to inspect after the fact and every future incident is a screenshot and a shrug. Send the logs to S3 (with the access controls and retention that customer content demands) and, for the interactive features, a filtered slice to CloudWatch Logs for quick searching. Then build the scorer: sample the logged summaries and run an LLM-as-a-judge pass that rates each against its source document for faithfulness, publish the pass rate and its inverse, the hallucination rate, as custom metrics, and alarm when the hallucination rate climbs. Scoring a sample against its sources is the mechanics of that pass. Tag every score with the prompt version, which gives you prompt effectiveness (pass rate against the tokens the version spends) and makes a prompt change that bought two points of accuracy for triple the context obvious at the moment it ships, plus response drift on any version whose pass rate slides while its text stays the same. Layer in the cheaper proxies alongside it: guardrail intervention rate, which the platform already emits, and user feedback wired into the app, so thumbs-down and edit-and-resend become signals you can graph. None of these is a perfect measure of correctness, but together they turn quality from an occasional manual spot-check into something that trends on a dashboard and pages when it regresses.

Publishing the built signals. Time-to-first-token, the hallucination rate and the business metrics all end up described as “a custom metric”, which skips over how they get there. The cheap default is the CloudWatch embedded metric format: the Lambda or service that already writes a structured log line for the interaction embeds the metric values in that same JSON, and CloudWatch extracts them into metrics on ingest. One write gives you the metric to alarm on and the log record to query later, with the interaction id in both, so a spike on the graph leads straight to the records behind it. Keep PutMetricData for the cases with no log line to ride on, an offline scoring job publishing a batch pass rate, for instance, and accept that it is a separate API call with its own cost and its own throttle.

Business impact. Once the operational three are in place, wire the application’s own events into the same dashboard: containment rate, escalation rate, edit-and-resend rate, abandonment and task completion, keyed by interaction id so they join to the invocation logs and the traces. This is what makes the token metrics arguable. A summariser that costs twice as much and halves the number of documents a person has to read through by hand is a good trade, and cost per invocation is the one view that cannot show it.

Worked example

The finance complaint is the quickest to close, and it shows how the free metrics and the built attribution combine. The team creates three application inference profiles over the same underlying model, one per feature, each tagged with the feature name, and points the chat panel, the summariser, and the inline helper at their respective profiles. Nothing about the model or the prompts changes; only the call path is now labelled.

Within a day the token-count metrics tell the story the aggregate could not. The chat panel and the inline helper sit flat. The overnight summariser’s InputTokenCount has roughly quadrupled over the month, tracking a change that let it pass entire documents into context instead of a trimmed extract. It was never looping and never misbehaving in any way an error metric would show; it was simply feeding the model four times the tokens per run, and on a per-token bill that is the whole doubling.

The fix is now a specific, scoped change to one feature (cap or pre-summarise the document before it hits the model, or move the batch onto provisioned throughput so its cost is a fixed line rather than a per-token one), and the alarm on InputTokenCount per profile means the next such creep pages the team instead of surfacing on an invoice five weeks later. The same tags that answered “which feature” also make the quality sampling per-feature, so the summariser that caused the cost scare is also the one whose faithfulness score now gets watched most closely.

What’s worth remembering

1. A production Bedrock app breaks in three directions (cost, latency, quality), and they live in three different places; one “is Bedrock up” dashboard will measure the easy two and miss the one that generates the worst complaints.
2. Cost is driven by tokens, not requests, so watch InputTokenCount and OutputTokenCount; an invocation count tells you almost nothing about spend.
3. For streaming features the number users feel is time-to-first-token, which Bedrock does not emit; instrument the stream client-side and publish it as a custom metric.
4. Model invocation logging captures full prompts and completions plus metadata to S3 and/or CloudWatch Logs, but it is off by default and must be explicitly enabled; you cannot analyse outputs you never kept.
5. Use tagged application inference profiles to attribute cost and usage to individual features; without them, on-demand spend collapses into one model line you cannot break down.
6. The platform has no metric for correctness or faithfulness, so quality and business impact are the signals you build: sample the logged completions for a hallucination rate, track prompt effectiveness by prompt version, and publish containment and escalation alongside the token counts.