The situation
A veterinary practice-management SaaS runs a visit-note drafting feature. The vet dictates while the animal is still on the table, an Amazon Bedrock model turns the dictation into a structured clinical note, and the vet edits and signs it before it goes on the patient record. Around nine hundred practices use it, and it drafts something close to forty thousand notes a week.
The team evaluates the feature properly. In one quarter it ran three comparisons with Amazon Bedrock Model Evaluations. In April, a cheaper candidate model against the incumbent on the same two-thousand-example set. In June, a rewritten prompt template against the one it replaced. In August, a fine-tuned candidate against the base model it was tuned from. Each job wrote per-record scores and a summary to Amazon S3. Each time, somebody pasted the summary into the team channel with a paragraph of commentary, three people reacted to it, and the channel moved on.
The other signals are all in place. CloudWatch carries invocation counts, input and output token counts, server-side latency, and throttles. Cost allocation tags split the Bedrock spend by feature, the way tagging a generative-AI workload sets it up. A DynamoDB table records, for every drafted note, whether the vet signed it unchanged, edited it, or threw it away and typed the note themselves.
The product owner has now asked the same two questions three months running. Which model is the feature on? And did quality move after the June prompt change? Both answers exist, in S3, in full detail. Producing them takes an engineer most of a morning of scrolling back through the channel and re-running queries by hand, which is why the third asking got the same treatment as the first.
What actually matters
Splitting a view by reader is settled ground for the operational side of a feature like this, and the split between the engineer, the product owner and the reviewer already covers how that goes. An evaluation result behaves differently from every signal in that split, in one way that decides the build: nobody is waiting for it. A latency graph gets opened because something went wrong and somebody went looking. A comparison score has to travel to a person who was not in the room when the job ran, who has no cue telling her it exists, and who is deciding what to fund three weeks later. Three comparisons have been produced here and none of them reached a decision, so what needs designing is delivery rather than display.
Refresh cadence is the second split, and it is the one that quietly ruins a shared panel. An evaluation job produces one result every few days at best, and often one per candidate, which might mean three points in a quarter. Invocation metrics produce a point a minute, forever. Put both on one time axis and the slow series reads as a fault. A flat line with long gaps, next to a dense one that moves, looks exactly like a broken emitter. The statistics a metric surface applies by default, an average over five minutes or a sum over an hour, mean nothing on a score that arrives twice a month. Sparse and dense series belong on different surfaces, and the split follows cadence rather than importance.
The third thing is where the results actually live. A Bedrock Model Evaluations job leaves its output as data at rest in S3: a per-record file with the score and the reasoning for every example, and a summary alongside it. Comparing April with June with August means reading three prefixes as one table. That makes the reporting choice a query choice before it is a visualisation choice, and the work in front of the team is a partitioning scheme, a table definition, and somewhere to run SQL. Once the scores are a table, the tagged cost export and the acceptance data in DynamoDB join to them on model version and date. Cost-performance analysis then stops being an exercise somebody does by hand. Token efficiency, the latency-to-quality ratio, and the business outcomes the feature exists to move become columns next to the scores rather than three separate investigations.
The last property is what the answer costs to keep alive once it exists. Licensing that scales with the audience rather than with the number of panels means a surface built for six named readers can be wrong for sixty. Somebody has to own the thing when a job’s output changes shape or a metric is renamed. And a report someone has to remember to ask for stops arriving, because the asking is what gets dropped when the quarter is busy. Automated reporting mechanisms are worth more here than a better chart: the numbers were never ugly, they never reached the person deciding.
What we’ll filter on
- Accumulation: does it hold every comparison the team has run as one series, or report one job at a time?
- Cadence fit: is it built for a point a minute, or for one point per evaluation job?
- Reach: does the answer arrive without anyone remembering to ask, and without an AWS console sign-in?
- Joins: can it put evaluation scores next to tagged cost and note acceptance in one view?
- Escalation: can a score crossing a threshold reach somebody, and how quickly?
- Cost and upkeep: what does the next reader cost, and who fixes it when a job’s output changes shape?
The landscape
The evaluation job’s own report card
Bedrock Model Evaluations renders a report for the job you just ran, in the console, with the per-metric summary and a sample of scored records. It is the right first read for the engineer who launched the job, and it needs no build. What it will not do is hold history: it reports on one job, so it cannot draw April against June, and it has no idea what the feature costs or how often a vet signed the draft. Reaching it needs a console role. Treat it as the place a result is first inspected rather than the place a result is published.
A CloudWatch dashboard fed by custom metrics
A Lambda that fires on evaluation job completion can publish the headline score as a custom metric with PutMetricData, dimensioned by model identifier and metric name, and a CloudWatch dashboard then plots it next to invocation latency and token counts. The gain is alarming: a score that drops below a threshold can raise an alarm on the same footing as an error rate, which is how a golden-set score becomes a deployment gate. The losses are the sparse-series problem above, no drill-down to the record that scored badly because that record is in S3 and not in the metric, no join to spend, and a console sign-in for anyone who wants to look.
Amazon Managed Grafana over CloudWatch and Athena
The multi-source panel set: one workspace over CloudWatch metrics and an Athena query at once, sign-in through IAM Identity Center or SAML rather than a console role, its own alerting, and per-active-user licensing that tracks the size of the audience. Against this problem its particular use is that a sparse Athena series and a dense CloudWatch series can live in one workspace on separate panels, under one login. It suits a platform team already running a workspace for other services and wanting the generative-AI panels beside them.
Amazon Quick Sight over Amazon Athena
Amazon Athena reads the evaluation output straight from S3 as a table registered in the AWS Glue Data Catalog, partitioned by job date and model version. The tagged cost export lands beside it, and a scheduled export of the acceptance table joins on note identifier. Amazon Quick Sight, the business-intelligence side of Amazon Quick, sits on top with datasets in SPICE, calculated fields for the derived ratios, and dashboards embedded in an internal portal, priced per reader rather than per panel. It is the only option here that can put a quality score, a token count, a latency figure, and a line of the bill in the same chart. It can also raise an alert: a reader sets a threshold on a KPI visual, or turns on anomaly detection over the series, and gets an email when the rule trips. That alert is evaluated when the dataset refreshes, so it is a note waiting in the morning rather than something that reaches an on-call rota while the drafting service is still degraded.
A scheduled digest built with EventBridge, Lambda, and Amazon SNS
An Amazon EventBridge schedule fires on the first of the month. A Lambda runs a handful of saved Athena queries, composes the numbers into a short message, and publishes it to an Amazon SNS topic. The product owner, the clinical lead, and the finance partner are subscribed to it. An EventBridge rule on evaluation job completion can send the same shape of message the moment a comparison finishes. It costs almost nothing per reader, it needs no sign-in anywhere, and it arrives without anyone remembering to ask. What it cannot do is let a reader follow a number anywhere: whatever is in the message is the whole of it, and the format is maintained as code rather than dragged around a canvas.
Evaluation
Side by side
| Surface | Refresh | Reader without a console role | Cost shape | Joins scores to cost and outcomes | Can alarm | Arrives unasked |
|---|---|---|---|---|---|---|
| Job report card in the console | Per job | ✗ | Included | ✗ | ✗ | ✗ |
| CloudWatch dashboard, custom metrics | Seconds to minutes | ✗ | Per metric and dashboard | ✗ | ✓ | ✗ |
| Amazon Managed Grafana | Near real time | ✓ (Identity Center) | Per active user | ✓ (via Athena) | ✓ | ✗ |
| Amazon Quick Sight over Athena | Scheduled, per job or daily | ✓ (embedded) | Per reader, plus data scanned | ✓ | ✓ (email, on refresh) | ✓ (scheduled email) |
| EventBridge, Lambda, and Amazon SNS digest | Monthly, or on job completion | ✓ (no sign-in) | Per invocation, negligible | ✓ (query-side) | ✗ | ✓ |
Read down the joins column and the alarm column together. The surface that can wake somebody sits on a metric namespace, and a metric namespace has no join, so it cannot tell you that the cheaper model held its score while halving the tokens. The surfaces that can do that join are querying a table on a schedule, and their alerting inherits the schedule: a Quick Sight threshold alert is a real mechanism, but it waits for the refresh and arrives by email. That is the right speed for a quality trend and the wrong speed for a degraded service. Two surfaces, chosen for two readers, beats one surface that half-serves both.
Matching the reader to the surface
The solution
Start by giving the evaluation output somewhere to accumulate. Every Bedrock Model Evaluations job writes to one bucket, under a prefix partitioned by job date and model version. The third comparison then lands beside the first two instead of somewhere a human has to remember. A Glue crawler or an explicit table definition registers the per-record output and the summary as two Athena tables. A view flattens the metrics a reader cares about into one row per job per model. From here the acceptance table exports nightly to S3 and joins on note identifier, and the cost and usage data joins on the feature tag and the date. That is a morning of work, and it converts an archive of job artefacts into something anybody can query.
The model comparison visualisations then go in Quick Sight, because that is the only surface where the scores can sit next to money and behaviour. The dashboard carries four panels. A per-metric comparison bar for each candidate against the incumbent on the same evaluation set, which answers whether the candidate was better. A trend of the headline metric across every job the team has run, annotated with what changed, which answers whether quality moved after a specific change. A cost-performance panel carrying three axes read together. Token efficiency, expressed as output tokens per accepted note, so a model that rambles is penalised for it. The latency-to-quality ratio, plotting median server-side latency against the quality score, so a candidate that scores half a point higher and takes two seconds longer shows up as the trade it is. And the business outcomes, which here means the share of drafts signed unchanged and the share thrown away. A fourth panel breaks all of it down by practice size, because a model that suits a large multi-vet practice can be worse for a single-vet one.
CloudWatch keeps the operational signals and takes exactly one number from the evaluation side. A Lambda subscribed to job-completion events publishes the headline quality score as a custom metric dimensioned by model version, and an alarm fires if it falls below the gate threshold. That single metric is worth publishing because it reaches the rota within a minute of the score landing, where the same threshold set in Quick Sight would wait for the next dataset refresh and arrive as an email. Everything else about the job stays in S3 where the join lives, and the CloudWatch dashboard stays what the operational dashboard is for: latency, tokens, throttles, errors.
The digest closes the loop that failed three times already. An EventBridge schedule runs on the first of the month, a Lambda executes four saved Athena queries, and Amazon SNS delivers a short message: the model currently serving traffic and since when, the headline quality score with its change on the previous month, spend for the month with cost per accepted note, and the share of drafts signed unchanged. It ends with a link into the Quick Sight dashboard for anyone who wants to go further. A second EventBridge rule on job state change sends the same shape of message when a comparison finishes, so a result reaches the product owner on the day it exists rather than the next time she thinks to ask. Quick Sight’s own scheduled email report covers the same ground for readers who want the rendered dashboard rather than the numbers.
Managed Grafana stays out for the same reason it stayed out of the operational build: it would do the job, and for one team with three named external readers it buys a per-active-user bill and a second alerting system for coverage CloudWatch and Quick Sight already give. A platform team already running a workspace across a dozen services decides it the other way, and should.
One more thing follows from the table existing. The same schema takes results from every comparison the team runs, not only from offline evaluation jobs. Multi-model evaluation, where three candidates are scored against one set in a single pass, writes one row per model per metric. A/B testing and canary testing of FMs routes a slice of live traffic to a candidate, the way switching models without shipping code arranges it, and writes rows keyed on the variant that served each note. Because those land in the same table as the offline scores, the trend panel shows offline and live evidence for a model version side by side, which is the comparison that decides a promotion.
Worked example
Which model are we on?
The product owner opens the dashboard on the first of the month, having already read the number in her inbox. The trend panel’s rightmost point is annotated with the model version, the date it took full traffic, and the job identifier of the comparison that cleared it. One click into the per-practice-size panel shows the same version serving every segment, so there is no half-finished rollout hiding behind an average. What took an engineer a morning is now a glance. It is a glance because the model version is a partition key on every row rather than a fact somebody remembered to type into a message.
Did quality move after the June change?
The trend panel shows the headline score rising four points at the June annotation and holding. The cost-performance panel underneath tells the fuller story: output tokens per accepted note fell nineteen per cent, because the rewritten prompt stopped the model restating the presenting complaint in the assessment section, and median latency fell with it. The share of drafts signed unchanged rose six points over the following fortnight, lagging the score change because practices came back from the change one rota at a time.
The August comparison reads differently on the same panels. The fine-tuned candidate scored a point and a half above the base model, and its latency-to-quality ratio was slightly worse: better notes, reliably slower. Set against the cost of maintaining a tuned model through the next base-model version, which is the concern promoting a fine-tuned model deals with, the product owner deferred it and asked for the comparison to be re-run after the next base release. That decision took eight minutes and left a screenshot attached to the meeting note. Not one line of it required an engineer to scroll back through a channel.
What’s worth remembering
- An evaluation result that lives in a chat thread has been produced but not delivered; naming the reader and the decision comes before choosing a visualisation tool.
- Evaluation scores arrive every few days and invocation metrics arrive every minute, so putting them on one panel makes the sparse series look broken and its averages meaningless.
- Bedrock Model Evaluations output is data at rest in S3, which makes the reporting choice a query choice: partition it, register it in the Glue Data Catalog, and query it with Athena.
- Quick Sight over Athena is the surface that can join scores to tagged cost and to business outcomes, which is what cost-performance analysis needs: token efficiency, the latency-to-quality ratio, and the outcome the feature exists to move.
- Keep CloudWatch for operational signals plus a single headline score that can raise an alarm, because a BI threshold alert waits for the next dataset refresh and arrives as an email.
- Automated reporting mechanisms beat a better chart: an EventBridge schedule, a Lambda running saved queries, and an Amazon SNS topic mean the trend arrives without anyone remembering to ask for it.