The situation
A logistics SaaS company has run a retrieval-backed assistant inside its product for about a year. It answers questions about shipments, rate cards, and customs paperwork, using an Amazon Bedrock model over a knowledge base that indexes each customer’s own documents. Roughly two hundred tenants use it. The team has been diligent about instrumentation, and the signals are all there.
CloudWatch carries the operational numbers: invocation counts, input and output token counts, server-side invocation latency, throttling, and the error rates of the Lambda functions behind the API. Model invocation logging writes every prompt, completion, and guardrail intervention to S3, partitioned by date. A DynamoDB table holds feedback events, one row per thumbs-up or thumbs-down with a reason code and the request ID that produced it. Amazon OpenSearch Service reports query latency and result counts for the vector index. Cost allocation tags split the Bedrock and OpenSearch spend by feature and by tenant, and Cost Explorer can show it.
In one week three people asked for a dashboard. The on-call engineer wants something that tells her at two in the morning whether the assistant is degraded, and wants to be paged before a customer notices. The product owner wants a monthly view: deflection rate, cost per resolved conversation, which tenants use it and which have quietly stopped. The compliance reviewer wants to demonstrate to an auditor that the conversations where a guardrail fired were reviewed, and to pull any single conversation end to end for the twelve months the retention policy covers. The team’s instinct was to open a blank CloudWatch dashboard and start adding widgets for all three.
What actually matters
Observability platforms get sold as one place to see everything, and that phrasing hides the split that decides this design. Complete visibility is a property of the signals you collect, not of the number of screens you put them on. This team already has the signals. What they lack is a decision about which reader each view is built for, and the three readers pull in different directions on four properties before any service gets named.
The first is refresh latency. The on-call view is worthless if it is an hour behind, because the whole reason to look at it is to catch a live degradation. The product view is worthless if it is expensive to keep live. Nobody makes a monthly decision on a number that changed in the last thirty seconds. Yesterday’s close of business is fine, and a scheduled refresh over a settled dataset is cheaper and steadier than a live query. The compliance view has no refresh cadence at all. It is answered on demand, against months of history, and the query may take a minute without anyone minding. Those three cadences map onto three different storage and query paths, and forcing them onto one path makes two of the three worse.
The second is who signs in. The on-call engineer already has a console role, so putting her view in the AWS console costs nothing extra. The product owner and the compliance reviewer do not have console access and should not be given an IAM role with console sign-in so they can read a chart. A view that a non-AWS reader can open, with its own sign-in and its own row-level restrictions, is a different requirement from a view for an engineer who is already authenticated into the account. Access model is an input to the choice, not something to bolt on afterwards.
The third is whether the view has to join across stores. Operational questions are answerable inside one metric namespace: latency is up, throttles are up, errors are up. Business questions are joins. Deflection rate needs the invocation log and the feedback events keyed on request ID. Cost per resolved conversation needs both of those and the tagged spend. Per-tenant usage needs the tenant ID carried on every record and grouped. A joined question wants a query engine over a catalogue of tables; a metric namespace has no join.
The fourth is what the view is for once it exists. A chart that can raise an alarm and a chart that summarises a quarter are different artefacts. Compliance monitoring is the case where a chart is the wrong output entirely. An auditor handed a bar chart of guardrail interventions will ask what the bars are made of, and the answer has to be a queryable record with a stated retention period. Charts compress, and compression is what you cannot hand over as evidence.
What we’ll filter on
- Refresh latency: how stale may the number be, seconds, a day, or answered on demand against history?
- Joins: must the view combine metrics, logs, feedback events, and billing data into one figure?
- Reader sign-in: can a reader without an AWS console role be given the view, with per-tenant restrictions?
- Cost shape: does adding a reader cost per user, per dashboard, per query scanned, or nothing?
- Alarmable: can a threshold on this view page someone at two in the morning?
- Evidence: does the underlying record survive months and come back queryable, with a retention policy attached?
The landscape
CloudWatch dashboards
The metrics Bedrock publishes arrive in CloudWatch with no work: invocations, input and output token counts, invocation latency, and the throttling and error counts. Custom metrics you emit from the application, such as client-perceived time-to-first-token or a rolling LLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. score, land in the same namespace and graph beside them. Because alarms live in the same service, a graph and a page are the same object viewed twice, which no other surface here can say. A Logs Insights widget puts a log query on the same dashboard, so a count of guardrail interventions can sit next to a latency graph.
The limits shape how you use it. A dashboard is scoped to one account and one region by default; CloudWatch cross-account observability fixes that by designating a monitoring account and linking source accounts, so one dashboard spans a multi-account deployment. Three dashboards with up to fifty metrics each are free, and beyond that there is a small monthly charge per dashboard. Every reader needs console sign-in.
Amazon Managed Grafana
A Managed Grafana workspace connects to several data sources at once and puts them in one panel set: CloudWatch metrics and logs, OpenSearch, Athena, Prometheus, X-Ray. That is the surface for a question whose answer lives in two stores, such as retrieval latency from OpenSearch plotted against model latency from CloudWatch on one time axis. Sign-in goes through IAM Identity Center or a SAML provider, so a reader gets a login without an AWS console role. Grafana has its own alerting, so panels here are alarmable too. Licensing is per active user per month, split into editor and viewer rates, which makes the cost scale with the audience rather than with the number of panels.
Amazon Quick Sight over Athena
Athena reads the invocation logs directly out of S3 as a partitioned table registered in the AWS Glue Data Catalog, and a scheduled export of the DynamoDB feedback table joins to it on request ID. Quick Sight, the business-intelligence part of Amazon Quick, sits on top: datasets held in SPICE and refreshed on a schedule, calculated fields for the derived business figures, and row-level security so an account manager sees only their tenants. Readers sign in to Quick Sight rather than the console, and the dashboards embed into an internal portal, so the product owner never touches AWS. Pricing is per reader per month with a session cap, or capacity pricing once the audience is large. Refresh is scheduled, so this surface is a day behind by design and nothing here will page anyone.
OpenSearch Dashboards
Where logs already land in OpenSearch, its bundled dashboards come free with the domain. They are strong at the things a metric store is weak at: free-text search across prompts and completions, aggregations over high-cardinality fields such as tenant ID, and anomaly detection on a time series. Fine-grained access control maps roles to IAM Identity Center or SAML identities, and index state management applies the retention policy. It has no view of the bill, so business figures that reconcile against spend are not its work.
AWS Chatbot
Not a screen at all. AWS Chatbot delivers CloudWatch alarm state changes and SNS notifications into a Slack or Microsoft Teams channel, with the alarm graph rendered inline and a set of permitted read-only commands available in the thread. For the on-call case this matters more than the dashboard does, because the dashboard is the thing you open after something told you to.
Evaluation
Side by side
| Surface | Refresh | Joins across stores | Non-console reader | Cost shape | Alarmable | Evidence-grade |
|---|---|---|---|---|---|---|
| CloudWatch dashboards | Near real time | ✗ (metrics only) | ✗ | Per dashboard, 3 free | ✓ | ✗ |
| Amazon Managed Grafana | Near real time | ✓ | ✓ (Identity Center) | Per active user | ✓ | ✗ |
| Quick Sight over Athena | Scheduled, daily | ✓ | ✓ (embedded) | Per reader, plus data scanned | ✗ | ✗ |
| OpenSearch Dashboards | Near real time | ✗ (one domain) | ✓ (SAML) | Domain capacity | ✓ (alerting plugin) | ✓ (with retention) |
| Athena query on the log store | On demand | ✓ | ✗ | Per terabyte scanned | ✗ | ✓ |
| AWS Chatbot | Push, on change | ✗ | ✓ (chat channel) | Free, pay for SNS | ✓ (delivery only) | ✗ |
Read down the alarmable column and the evidence column and they barely overlap. A surface built to notice a change in the last minute is built on aggregated metrics, and aggregation is what destroys the individual record an auditor wants. A surface built to return that record months later is built on object storage and a query engine, and neither will page anyone. Trying to satisfy both from one place produces something that is slow to alarm and thin as evidence.
Routing the signals
The solution
Give each reader their own surface, built on the shared signal set, and stop trying to make one screen do three jobs.
Operations: CloudWatch, alarms, and a chat channel
The on-call surface is operational metric dashboards in CloudWatch, one per feature, holding the numbers that move when the assistant degrades. That means p95 invocation latency, client-emitted time-to-first-token, throttling count, Lambda error rate, OpenSearch query latency, and a Logs Insights widget counting guardrail interventions in the last hour. Every graph that matters carries an alarm, and a composite alarm rolls the individual ones into one “assistant degraded” state so the channel gets one message rather than six. AWS Chatbot delivers that state into the on-call Slack channel with the graph inline, which is where the engineer actually finds out. The dashboard is what she opens next. If the feature spans accounts, cross-account observability links them into a monitoring account so this stays one dashboard. What already exists behind this view is covered in the signals a production Bedrock app emits, and the per-request detail behind a slow trace in tracing an agent’s decisions.
Business: Quick Sight over Athena
The product owner’s view is business impact metrics with custom dashboards, and its inputs are the joins nothing else here can do. Athena reads the invocation logs from S3 as a partitioned Glue table; a nightly export of the DynamoDB feedback table lands beside it; the tagged spend, prepared the way cost attribution and tagging sets it up, joins on tenant and feature. Quick Sight builds business impact visualisations on top: deflection rate, cost per resolved conversation, tokens per tenant, and the trend in negative feedback that the feedback loop is already collecting. SPICE refreshes nightly, row-level security restricts each account manager to their own tenants, and the dashboard embeds into the internal portal so nobody needs an AWS login. Two signal families feed this view and both have to be built rather than collected. User interaction tracking covers which features were opened, how many turns a conversation ran, and where people abandoned it. Model behavior pattern tracking covers the drift in response length, refusal rate, and judge scores over weeks. Neither is emitted by the platform.
Compliance: a query and a retention policy
Compliance monitoring is answered with a query, not a chart. The invocation logs stay in S3 under a lifecycle policy that holds them for the twelve months the retention standard names. S3 Object Lock in governance mode stops anything being deleted early, and a KMS key whose policy the security team owns covers the store. CloudTrail records the control-plane changes: who edited a guardrail, who changed a model ID, who touched the log configuration. Forensic traceability and audit logging come from saved Athena queries, one per control, each returning rows rather than a picture. Every conversation where a guardrail intervened in the quarter, joined to whether a reviewer signed it off. Every invocation against a model outside the approved list. Every request for one named tenant across the full retention window. The reviewer gets a signed CSV export and the query text that produced it. This is the evidence side of making a Bedrock app audit-ready, and a Quick Sight bar chart would be a worse answer than a plain table with a query attached.
Where Managed Grafana fits
Managed Grafana is worth adding when a platform team runs several generative-AI features and wants one panel set crossing CloudWatch, OpenSearch, and Athena. It also fits when the readers of that view need a sign-in but not a console role. For a single feature with three named readers it adds a per-user bill and a second alerting system for something the three surfaces above already cover. Reach for it when the audience is a platform team, not when it is one engineer.
Worked example
At 07:38 on a Tuesday the composite alarm flips and AWS Chatbot posts it into the on-call channel with the latency graph attached. The engineer opens the CloudWatch dashboard: p95 invocation latency is flat, but OpenSearch query latency has tripled and time-to-first-token has gone with it. Throttling is unchanged, so this is not a quota problem. She checks the index and finds a reindex job from the previous evening still running against the same domain. She throttles the job, latency returns, and the alarm clears at 08:05. Nothing in the product or compliance surfaces moved, and neither should have.
On Thursday the product owner opens the Quick Sight dashboard and sees deflection rate for one large tenant down eleven points across a fortnight, with negative feedback up on a single reason code. Drilling into the per-tenant panel shows the drop starts on the day that tenant uploaded a new rate card. This is a question the operations dashboard could never have raised, because every invocation succeeded, quickly, at normal cost. The answer is wrong, not slow.
At quarter end the compliance reviewer runs the saved query for guardrail interventions. It returns 41 conversations, joined against the review table to show 39 signed off and two outstanding. For one of those two she takes the request ID, runs the trace query against the retained log, and gets the prompt, the retrieved passages, the completion, and the guardrail decision as rows. She exports the result, attaches the query text, and the auditor has the record rather than a summary of it.
What’s worth remembering
- Complete visibility is a property of the signals you collect; how many dashboards you build is a separate decision, made per reader.
- Refresh cadence, cross-store joins, reader sign-in, and evidence quality pull in different directions, so sort the audience by those four before naming a service.
- CloudWatch dashboards are alarmable and near real time, which is exactly what on-call needs and exactly what a monthly business review does not.
- Business figures such as deflection rate and cost per conversation are joins across the invocation log, the feedback store, and tagged spend, so they belong on a query engine and a scheduled BI refresh.
- Compliance wants the individual record with a stated retention period and a query that reproduces it, and a chart aggregates away the one thing the auditor came for.
- Amazon Managed Grafana is the multi-source panel set for a platform team; for a single feature with three named readers it adds per-user cost for coverage you already have.