Exam Room · Advanced Generative AI Developer

Pop Quiz: Seeing Into a Production Bedrock App

· 4 min read

Exam-style

A production Bedrock application needs alerting when invocation latency or throttling climbs, and the compliance team separately wants to inspect the exact prompt and response behind a specific customer complaint. What combination covers both needs?

Reveal the answer

B. CloudWatch metrics and alarms for latency and throttling, plus model invocation logging for the content

The two needs take two mechanisms. Bedrock publishes Invocations, InvocationLatency, InvocationThrottles, InputTokenCount and OutputTokenCount to CloudWatch under the AWS/Bedrock namespace, dimensioned by ModelId, and an alarm evaluates those. Model invocation logging is the separate feature that records request body, response body and invocation metadata to S3, to CloudWatch Logs, or to both. It is off by default, and once enabled it covers every bedrock-runtime call in that account and Region; bodies over 100 KB and binary data go to S3 rather than inline. CloudTrail logs InvokeModel and Converse as management events, naming the caller, the time and the model ID, with no prompt, no response and no latency, and its data event selectors cover agents, knowledge bases, flows and guardrails rather than invocation payloads. X-Ray annotation values are strings of up to 250 characters, so a prompt would have to sit in unindexed metadata inside a 64 KB segment document, and the SDK debug route scatters sensitive content wherever a developer sent it. Config records the configuration of Bedrock resources such as guardrails and knowledge bases, and has no record of an individual invocation.

Generative AI Development · part of The Exam Room

Q. What gives you operational visibility into a production Bedrock app?

A. CloudWatch metrics in the AWS/Bedrock namespace: Invocations, InvocationLatency, InvocationThrottles, and input and output token counts, with alarms on them. Model invocation logging, off by default, captures the prompt and the response. CloudTrail records the call and the model ID, not the payload.

Why? Rate metrics and payload records come from separate mechanisms, so a production app turns on both.

These posts are LLM-aided. Backbone, original writing, and structure by Craig. Research and editing by Craig + LLM. Proof-reading by Craig.