Monitoring a Production Bedrock App

The situation

A product team has shipped a customer-facing assistant on Amazon Bedrock. It runs a single Claude model behind three features: a chat panel that streams answers, a document-summarisation job that runs in batches overnight, and an inline “explain this” helper embedded in the app. Traffic has grown from a demo to real load, and three separate complaints have landed in the same week.

Finance has noticed the Bedrock line on the AWS bill more than doubled month on month, and nobody can say which of the three features is responsible or whether one of them is looping. Support has forwarded a handful of screenshots where the streaming chat sat blank for eight or nine seconds before any text appeared, and users assumed it had hung. And a manual spot-check turned up two summaries that confidently invented figures that were not in the source document, which is the kind of thing that erodes trust faster than a slow response ever will.

The team has CloudWatch switched on and can see that “Bedrock is being called a lot”. What they do not have is a way to tell cost, latency, and quality apart, attribute any of them to a feature, or catch the next regression before a customer does. The underlying question is which signals Bedrock gives you, which you have to construct, and where each one lives.

What actually matters

The first thing worth naming is that these are three different problems with three different homes, and conflating them is why the team is stuck. Cost and latency and throttling are operational metrics the platform emits about itself. Quality is a property of the content, and the platform has no opinion about content. Any monitoring design that treats “is Bedrock healthy” as one dashboard will measure the two easy things and silently skip the one that actually generated the complaint about invented figures.

Cost on Bedrock is driven by tokens, not requests, so a request count tells you almost nothing about spend. Two calls with the same invocation count can differ tenfold in cost because one stuffed a whole document into the context and the other asked a one-line question. That means the useful cost signal is input and output token counts, and the useful cost question is per-feature: which of the three features is burning the budget, and is any of them regressing toward longer prompts or runaway output. Attribution matters more than the aggregate, because you cannot fix a bill you cannot break down.

Latency has a shape that a single average hides, and streaming makes this sharper. For a streaming feature the number the user actually feels is time-to-first-token, the wait before anything appears, which is a different quantity from total generation time. A response that streams for six seconds but starts in under one feels fast; a response that starts after eight seconds feels broken even if it finishes sooner overall. Server-side invocation latency and client-perceived time-to-first-token are both worth having, and they are not the same measurement.

Throughput and throttling are the capacity story. Bedrock enforces account-level and model-level quotas, and on-demand traffic that pushes past them comes back as throttling errors rather than a slowdown. If throttles are climbing you are leaving requests on the floor, and the fix is a capacity decision (request a quota increase, or move the steady load onto provisioned throughputProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. ), not a code change. This signal has to be visible before customers feel it as failures.

Quality is the one the platform cannot see. Bedrock will happily report that an invocation succeeded, returned 400 tokens, and took 900 milliseconds, while those 400 tokens contain a fabricated number. There is no server-side metric for correctness, faithfulnessFaithfulnessWhether every claim in an answer is actually supported by the source it was given, regardless of whether it happens to be true. , or tone, because those are judgements about meaning. So quality monitoring is something you build on top: capture the actual prompts and completions, sample them, and score the sample by some method that understands content, whether that is a second model acting as a judge, a human reviewer, or a proxy signal like how often a guardrail had to intervene or how often users reacted badly. The distinction that runs through the whole design is free metrics versus built signal: the platform gives you the operational numbers, and you construct the quality one or you do not have it.

What we’ll filter on

1. Is the signal emitted by the platform, or does it have to be constructed? Cost, latency, and throttling are emitted; quality is constructed.
2. Can it be attributed to a specific feature? Per-feature cost and error breakdown is worth more than a single aggregate.
3. Does it capture the user-perceived shape, not just a server-side average? Time-to-first-token for streaming, percentiles rather than means.
4. Does it support alarms, so a regression pages someone instead of waiting for a complaint?
5. What does it cost to run, in storage and in review effort? Full-payload logging and human scoring are not free.

The observability landscape

CloudWatch metrics from Bedrock. Bedrock publishes runtime metrics to the AWS/Bedrock namespace automatically, at no extra cost, dimensioned by model. The ones that carry the load here are Invocations (call count), InvocationLatency (server-side processing time), InputTokenCount and OutputTokenCount (the token volumes that drive both cost and latency), and the error and throttle counters InvocationClientErrors, InvocationServerErrors, and InvocationThrottles. These are standard CloudWatch metrics, so you can graph them, take percentiles rather than averages, and set alarms on them. They tell you how much, how fast, and how often it failed. They tell you nothing about what was in the request or how good the answer was.

CloudWatch alarms and dashboards. On top of those metrics you build the operational layer: an alarm on rising InvocationThrottles so capacity pressure pages before it becomes user-visible failures, an alarm on p99 InvocationLatency, an alarm on an OutputTokenCount sum that jumps past its normal band (the cheapest early warning for a feature that has started looping or ballooning its output). A dashboard puts token counts, latency percentiles, and error rates side by side so the three failure directions are legible at a glance. This is all standard CloudWatch, nothing Bedrock-specific beyond the metric names.

Bedrock model invocation logging. This is the piece you have to turn on deliberately; it is off by default. Once enabled in the Bedrock settings for the account and region, it captures the full request and response for invocations (the complete prompts and completions) along with metadata, and delivers them to Amazon S3, to Amazon CloudWatch Logs, or to both. Large payloads and any image or embedding data are written to an S3 bucket you nominate. This is the record you need for debugging a specific bad answer, for audit, and above all for offline quality analysis, because you cannot score outputs you never kept. It is also the sensitive one: full prompts and completions can contain customer data, so the log destination needs the same access controls and retention thinking as any other store of user content.

Application inference profiles for cost attribution. An application inference profile wraps a foundation model with your own tags, and calls made through the profile carry those tags into cost and usage tracking. That is the clean way to answer “which feature spent the money”: give the chat panel, the summariser, and the inline helper each their own tagged profile, and the token spend splits by feature in Cost Explorer and in per-profile CloudWatch metrics instead of collapsing into one undifferentiated model line. Without something like this, on-demand spend is attributed to the model, not to the feature that made the call.

Bedrock Guardrails signals. If the app runs Bedrock Guardrails, the rate at which the guardrail intervenes (blocking or masking content, on input or output) is a genuine quality-adjacent signal that the platform does surface. A climbing intervention rate says either the inputs are getting more adversarial or the model is drifting toward output the policy rejects, and either way it is worth an alarm. It is a proxy for quality, not a measure of correctness, but it is one you get without building a scorer.

The quality layer you build. Nothing above measures whether the summary was faithful to the document. That layer is yours to construct on top of invocation logging: sample the captured completions on some cadence, and score the sample. The score can come from an LLM-as-a-judge pass (a second model prompted to rate faithfulness or correctness against the source), from human review of a small sample, or from user feedback signals wired into the app (thumbs up and down, edit-and-resend, abandonment). Whichever you pick, it runs offline against the logs, it works on a sample rather than every call because scoring is not free, and it is the only thing in the whole design that actually looks at meaning.

Side by side

Signal	Source	Emitted or built	Per-feature attribution	Alarmable	Catches the invented figure
Invocation count	CloudWatch Invocations	Emitted (free)	Via tagged inference profile	✓	✗
Token counts (cost)	CloudWatch InputTokenCount / OutputTokenCount	Emitted (free)	Via tagged inference profile	✓	✗
Server latency	CloudWatch InvocationLatency	Emitted (free)	By model dimension	✓	✗
Time-to-first-token	Client instrumentation on the stream	Built	By feature (you own it)	✓ (custom metric)	✗
Throttling	CloudWatch InvocationThrottles	Emitted (free)	By model dimension	✓	✗
Full prompts and completions	Model invocation logging to S3 / CloudWatch Logs	Emitted, but off by default	Yes, in the log record	✗ (it is a record, not a metric)	Only if you read or score it
Guardrail intervention rate	Bedrock Guardrails	Emitted (free)	By guardrail	✓	Partly (policy hits only)
Faithfulness / correctness	LLM-as-a-judge, human review, user feedback	Built on top of logs	Yes, by design	✓ (on the derived score)	✓

The picks in depth

Cost. Turn the aggregate into a per-feature breakdown before doing anything else, because the finance complaint cannot be answered otherwise. Give each of the three features its own application inference profile, tagged, so spend splits by feature in Cost Explorer and the token-count metrics carry the tag. Then watch the token counts, not the invocation count, since tokens are what the bill is made of. A summariser that has crept from 2,000-token to 8,000-token inputs shows up as a rising InputTokenCount long before the monthly bill lands, and a chat feature that has started producing 3,000-token rambles shows up in OutputTokenCount. Alarm on the sums stepping outside their normal band; that single alarm is the earliest, cheapest catch for both a runaway loop and a quiet prompt-size regression.

Latency. Measure it as the user feels it, which for the streaming chat means time-to-first-token, and that is not something Bedrock hands you as a metric. When you call the streaming API, timestamp the request and timestamp the first content chunk off the stream, and publish the difference as a custom CloudWatch metric. Keep the server-side InvocationLatency too for the batch summariser, where total time is what matters and there is no user waiting on a first token. Track both as percentiles, because a p50 that looks fine can hide a p99 that is generating the support screenshots. Alarm on p99 time-to-first-token for the interactive features and on total latency for the batch one.

Throughput and throttling. Put InvocationThrottles on a dashboard and an alarm from day one, because throttles are lost requests, not slow ones, and they are invisible in a latency graph. If they climb under normal load you have a capacity decision to make: request a service quota increase for the model, or move the predictable, steady traffic (the overnight batch summariser is the obvious candidate) onto provisioned throughput so it does not compete with the interactive features for on-demand capacity. This is a knob-turning fix, and the alarm is what tells you to reach for the knob.

Quality. This is the one the team does not have at all, and it is the reason a fabricated figure reached a customer. Start by enabling model invocation logging, because without the captured completions there is nothing to inspect after the fact and every future incident is a screenshot and a shrug. Send the logs to S3 (with the access controls and retention that customer content demands) and, for the interactive features, a filtered slice to CloudWatch Logs for quick searching. Then build the scorer: sample the logged summaries and run an LLM-as-a-judge pass that rates each against its source document for faithfulness, publish the pass rate as a custom metric, and alarm when it drops. Layer in the cheaper proxies alongside it: guardrail intervention rate, which the platform already emits, and user feedback wired into the app, so thumbs-down and edit-and-resend become signals you can graph. None of these is a perfect measure of correctness, but together they turn quality from an occasional manual spot-check into something that trends on a dashboard and pages when it regresses.

A worked example: attributing the doubled bill

The finance complaint is the quickest to close, and it shows how the free metrics and the built attribution combine. The team creates three application inference profiles over the same underlying model, one per feature, each tagged with the feature name, and points the chat panel, the summariser, and the inline helper at their respective profiles. Nothing about the model or the prompts changes; only the call path is now labelled.

Within a day the token-count metrics tell the story the aggregate could not. The chat panel and the inline helper sit flat. The overnight summariser’s InputTokenCount has roughly quadrupled over the month, tracking a change that let it pass entire documents into context instead of a trimmed extract. It was never looping and never misbehaving in any way an error metric would show; it was simply feeding the model four times the tokens per run, and on a per-token bill that is the whole doubling.

The fix is now a specific, scoped change to one feature (cap or pre-summarise the document before it hits the model, or move the batch onto provisioned throughput so its cost is a fixed line rather than a per-token one), and the alarm on InputTokenCount per profile means the next such creep pages the team instead of surfacing on an invoice five weeks later. The same tags that answered “which feature” also make the quality sampling per-feature, so the summariser that caused the cost scare is also the one whose faithfulness score now gets watched most closely.

What’s worth remembering

1. A production Bedrock app breaks in three directions (cost, latency, quality), and they live in three different places; one “is Bedrock up” dashboard will measure the easy two and miss the one that generates the worst complaints.
2. Cost is driven by tokens, not requests, so watch InputTokenCount and OutputTokenCount; an invocation count tells you almost nothing about spend.
3. Bedrock publishes Invocations, InvocationLatency, InputTokenCount, OutputTokenCount, InvocationClientErrors, InvocationServerErrors, and InvocationThrottles to CloudWatch automatically and for free; graph them as percentiles and alarm on them.
4. For streaming features the number users feel is time-to-first-token, which Bedrock does not emit; instrument the stream client-side and publish it as a custom metric.
5. Throttling is lost requests, not slow ones, so it needs its own alarm; the fix is a capacity decision (quota increase or provisioned throughput), not a code change.
6. Model invocation logging captures full prompts and completions plus metadata to S3 and/or CloudWatch Logs, but it is off by default and must be explicitly enabled; you cannot analyse outputs you never kept.
7. Those logs contain customer content, so the destination needs the same access controls and retention as any other store of sensitive data.
8. Use tagged application inference profiles to attribute cost and usage to individual features; without them, on-demand spend collapses into one model line you cannot break down.
9. The platform has no metric for correctness or faithfulness, so quality is the signal you build: sample the logged completions and score them with an LLM-as-a-judge pass, human review, guardrail intervention rate, or user feedback.
10. Quality scoring runs offline on a sample rather than on every call, because it is the one part of the design that is not free; turn its pass rate into a metric and alarm on the drop.