The situation
A veterinary practice-management SaaS runs a visit-note drafting feature. The vet dictates while the animal is still on the table, an Amazon Bedrock model turns the dictation into a structured clinical note, and the vet edits and signs it before it goes on the patient record. Around nine hundred practices use it, and it drafts something close to forty thousand notes a week.
The team evaluates the feature properly. In one quarter it ran three comparisons with Amazon Bedrock model evaluation, using a judge model to score the responses. In April, a cheaper candidate model against the incumbent on the same two-thousand-example set. In June, a rewritten prompt template against the one it replaced. In August, a fine-tuned candidate against the base model it was tuned from. An automated evaluation job scores one model, so each comparison was a pair of jobs read side by side. Every job wrote its per-record scores, with the judge’s explanation of each one, to Amazon S3 as JSON Lines. Each time, somebody pasted the headline numbers into the team channel with a paragraph of commentary, three people reacted to it, and the channel moved on.
The other signals are all in place. CloudWatch carries invocation counts, input and output token counts, server-side latency, and throttles. Cost allocation tags split the Bedrock spend by feature, the way tagging a generative-AI workload sets it up. A DynamoDB table records, for every drafted note, whether the vet signed it unchanged, edited it, or threw it away and typed the note themselves.
The product owner has now asked the same two questions three months running. Which model is the feature on? And did quality move after the June prompt change? Both answers exist, in S3, in full detail. Producing them takes an engineer most of a morning of scrolling back through the channel and re-running queries by hand, which is why the third asking got the same treatment as the first.
What actually matters
Splitting a view by reader is settled ground for the operational side of a feature like this, and the split between the engineer, the product owner and the reviewer already covers how that goes. An evaluation result behaves differently from every signal in that split, in one way that shapes the build: nobody is waiting for it. A latency graph gets opened because something went wrong and somebody went looking. A comparison score has to travel to a person who was not in the room when the job ran, who has no cue telling her it exists, and who is deciding what to fund three weeks later. Three comparisons have been produced here and none of them reached a decision, so what needs designing is delivery rather than display.
Refresh cadence is the second split, and it is the one that ruins a shared panel. An evaluation job produces one result every few days at best, and often one per candidate, which might mean three points in a quarter. Invocation metrics produce a point a minute, forever. Put both on one time axis and the slow series reads as a fault. A flat line with long gaps, next to a dense one that moves, looks exactly like a broken emitter. The statistics a metric surface applies by default, an average over five minutes or a sum over an hour, mean nothing on a score that arrives twice a month. Sparse and dense series belong on different surfaces, and the split follows cadence rather than importance.
The third thing is where the results actually live. A model evaluation job leaves its output as data at rest in S3: a JSON Lines file per metric and dataset, carrying the score, the input record and the model’s response for every example, with the judge’s explanation where a judge model did the scoring. Comparing April with June with August means reading six job prefixes as one table. That makes the reporting choice a query choice before it is a visualisation choice, and the work in front of the team is a partitioning scheme, a table definition, and somewhere to run SQL. Once the scores are a table, the tagged cost export and the acceptance data in DynamoDB join to them on model version and date. Cost-performance analysis then stops being an exercise somebody does by hand. Token efficiency, the latency-to-quality ratio, and the business outcomes the feature exists to move become columns next to the scores rather than three separate investigations.
The last property is what keeping the answer alive takes once it exists. Licensing that scales with the audience rather than with the number of panels means a surface built for six named readers can be wrong for sixty. Somebody has to own the thing when a job’s output changes shape or a metric is renamed. And a report someone has to remember to ask for stops arriving, because the asking is what gets dropped when the quarter is busy. Automated reporting mechanisms do more here than a better chart. The numbers were never ugly; they never reached the person deciding.
What we’ll filter on
- Accumulation: does it hold every comparison the team has run as one series, or report one job at a time?
- Cadence fit: is it built for a point a minute, or for one point per evaluation job?
- Reach: does the answer arrive without anyone remembering to ask, and without an AWS console sign-in?
- Joins: can it put evaluation scores next to tagged cost and note acceptance in one view?
- Escalation: can a score crossing a threshold reach somebody, and how quickly?
- Cost and upkeep: what does the next reader cost, and who fixes it when a job’s output changes shape?
The landscape
The evaluation job’s own report card
Bedrock renders a report for the job you just ran, in the console, with the per-metric summary card and the judge’s explanations for the first few prompts in the dataset. It is the right first read for the engineer who launched the job, and it needs no build. What it will not do is hold history: it reports on one job, so it cannot draw April against June, and it carries nothing about what the feature costs or how often a vet signed the draft. Reaching it needs a console role. Treat it as the place a result is first inspected rather than the place a result is published.
A CloudWatch dashboard fed by custom metrics
Bedrock sends no EventBridge event when an evaluation job finishes. The job state changes it does emit cover model customisation and batch inference, so the trigger has to come from somewhere else: the results landing in S3. With EventBridge notifications turned on for the bucket, a Lambda on the Object Created event for the output prefix publishes the headline score as a custom metric with PutMetricData, dimensioned by model identifier and metric name, and a CloudWatch dashboard then plots it next to invocation latency and token counts. The gain is alarming: a score that drops below a threshold can raise an alarm on the same footing as an error rate, which is how a golden-set score becomes a deployment gate. The losses are the sparse-series problem above, no drill-down to the record that scored badly because that record is in S3 and not in the metric, no join to spend, and a console sign-in for anyone reading it.
Amazon Managed Grafana over CloudWatch and Athena
The multi-source panel set: one workspace over CloudWatch metrics and an Athena query at once, sign-in through IAM Identity Center or SAML rather than a console role, its own alerting, and per-active-user licensing that tracks the size of the audience. Against this problem its particular use is that a sparse Athena series and a dense CloudWatch series can live in one workspace on separate panels, under one login. It suits a platform team already running a workspace for other services and wanting the generative-AI panels beside them.
Amazon Quick Sight over Amazon Athena
Amazon Athena reads the evaluation output straight from S3 as a table registered in the AWS Glue Data Catalog, partitioned by job date and model version. The tagged cost export lands beside it, and a scheduled export of the acceptance table joins on note identifier. Amazon Quick Sight, the business-intelligence side of Amazon Quick, sits on top with datasets in SPICE, calculated fields for the derived ratios, and dashboards embedded in an internal portal, priced per reader rather than per panel. Because the join happens in Athena rather than across panels, a quality score, a token count, a latency figure and a line of the bill can share one chart. It can also raise an alert: on a KPI, gauge, table or pivot visual a reader sets a threshold and gets an email when the value crosses it. Threshold alerts are an Enterprise edition feature, they cannot be created from an embedded copy of a dashboard, and on a SPICE dataset they are checked after each successful refresh. That makes the alert a note waiting in the morning rather than something that reaches an on-call rota while the drafting service is still degraded.
A scheduled digest built with EventBridge, Lambda, and Amazon SNS
An Amazon EventBridge schedule fires on the first of the month. A Lambda runs a handful of saved Athena queries, composes the numbers into a short message, and publishes it to an Amazon SNS topic. The product owner, the clinical lead, and the finance partner are subscribed to it. An EventBridge rule on the Object Created notification for the output prefix can send the same shape of message the moment a job’s results land. It costs almost nothing per reader, it needs no sign-in anywhere, and it arrives without anyone remembering to ask. What it cannot do is let a reader follow a number anywhere: whatever is in the message is the whole of it, and the format is maintained as code rather than dragged around a canvas.
Evaluation
Side by side
| Surface | Refresh | Reader without a console role | Cost shape | Joins scores to cost and outcomes | Can alarm | Arrives unasked |
|---|---|---|---|---|---|---|
| Job report card in the console | Per job | ✗ | Included | ✗ | ✗ | ✗ |
| CloudWatch dashboard, custom metrics | Seconds to minutes | ✗ | Per metric and dashboard | ✗ | ✓ | ✗ |
| Amazon Managed Grafana | Near real time | ✓ (Identity Center) | Per active user | ✓ (via Athena) | ✓ | ✗ |
| Amazon Quick Sight over Athena | SPICE refresh, per job or daily | ✓ (embedded) | Per reader, plus data scanned | ✓ | ✓ (email, on refresh) | ✓ (scheduled email) |
| EventBridge, Lambda, and Amazon SNS digest | Monthly, or on results landing | ✓ (no sign-in) | Per invocation, negligible | ✓ (query-side) | ✗ | ✓ |
Read down the joins column and the alarm column together. The surface that can wake somebody sits on a metric namespace, and a metric namespace has no join, so it cannot show that the cheaper model held its score while halving the tokens. The surfaces that can do that join are querying a table on a schedule, and their alerting inherits the schedule: a Quick Sight threshold alert is a real mechanism, but it waits for the refresh and arrives by email. That is the right speed for a quality trend and the wrong speed for a degraded service. Two surfaces, chosen for two readers, beats one surface that half-serves both.
Matching the reader to the surface
The solution
Start by giving the evaluation output somewhere to accumulate. Set every job’s output URI under one bucket, in a Hive-style prefix the team controls: job date, then model version. Bedrock writes its own job-name and job-uuid tree beneath that, so the partition keys stay where a query can read them. The third comparison then lands beside the first two instead of somewhere a human has to remember. A Glue crawler or an explicit table definition registers the per-record JSON Lines output as an Athena table, and a view aggregates it into one row per job per model, carrying the metrics a reader cares about. From here the acceptance table exports nightly to S3 and joins on note identifier, and the cost and usage data joins on the feature tag and the date. That is a morning of work, and it converts an archive of job artefacts into something anybody can query.
The model comparison visualisations then go in Quick Sight, where the scores sit next to money and behaviour with the join done in the query rather than across panels. The dashboard carries four panels. A per-metric comparison bar for each candidate against the incumbent on the same evaluation set, which answers whether the candidate was better. A trend of the headline metric across every job the team has run, annotated with what changed, which answers whether quality moved after a specific change. A cost-performance panel carrying three axes read together. Token efficiency, expressed as output tokens per accepted note, so a model producing longer drafts for the same result shows up as the more expensive one. The latency-to-quality ratio, plotting median server-side latency against the quality score, so a candidate that scores half a point higher and takes two seconds longer shows up as the trade it is. And the business outcomes, which here means the share of drafts signed unchanged and the share thrown away. A fourth panel breaks all of it down by practice size, because a model that suits a large multi-vet practice can be worse for a single-vet one.
CloudWatch keeps the operational signals and takes exactly one number from the evaluation side. A Lambda triggered when a job’s results land in the output prefix publishes the headline quality score as a custom metric dimensioned by model version, and an alarm fires if it falls below the gate threshold. That single metric is worth publishing because it reaches the rota within a minute of the score landing, where the same threshold set in Quick Sight would wait for the next SPICE refresh and arrive as an email. Everything else about the job stays in S3 where the join lives, and the CloudWatch dashboard stays what the operational dashboard is for: latency, tokens, throttles, errors.
The digest closes the loop that failed three times already. An EventBridge schedule runs on the first of the month, a Lambda executes four saved Athena queries, and Amazon SNS delivers a short message: the model currently serving traffic and since when, the headline quality score with its change on the previous month, spend for the month with cost per accepted note, and the share of drafts signed unchanged. It ends with a link into the Quick Sight dashboard for anyone reading further. A second EventBridge rule, on the Object Created notification for the output prefix, sends the same shape of message when a job’s results land, so a result reaches the product owner on the day it exists rather than the next time she thinks to ask. Quick Sight’s own scheduled email report covers the same ground for readers who prefer the rendered dashboard, though every recipient of one has to be in the Quick subscription with the dashboard shared to them. The SNS route has no such condition, which is why the finance partner is on it.
Managed Grafana stays out for the same reason it stayed out of the operational build: it would do the job, and for one team with three named external readers it adds a per-active-user bill and a second alerting system for coverage CloudWatch and Quick Sight already give. A platform team already running a workspace across a dozen services decides it the other way, and should.
One more thing follows from the table existing. The same schema takes results from every comparison the team runs, not only from offline evaluation jobs. An automated job scores one model and a human-worker job takes two, so putting three candidates against one set means three jobs, each writing its own rows into the same table rather than into three separate reports. A/B testing and canary testing of FMs routes a slice of live traffic to a candidate, the way switching models without shipping code arranges it, and writes rows keyed on the variant that served each note. Because those land in the same table as the offline scores, the trend panel shows offline and live evidence for a model version side by side, which is the comparison a promotion turns on.
Worked example
Which model are we on?
The product owner opens the dashboard on the first of the month, having already read the number in her inbox. The trend panel’s rightmost point is annotated with the model version, the date it took full traffic, and the job identifier of the comparison that cleared it. One click into the per-practice-size panel shows the same version serving every segment, so there is no half-finished rollout hiding behind an average. What took an engineer a morning is now a glance. It is a glance because the model version is a partition key on every row rather than a fact somebody remembered to type into a message.
Did quality move after the June change?
The trend panel shows the headline score rising four points at the June annotation and holding. The cost-performance panel underneath tells the fuller story: output tokens per accepted note fell nineteen per cent, because the rewritten prompt stopped the model restating the presenting complaint in the assessment section, and median latency fell with it. The share of drafts signed unchanged rose six points over the following fortnight, lagging the score change because practices came back from the change one rota at a time.
The August comparison reads differently on the same panels. The fine-tuned candidate scored a point and a half above the base model, and its latency-to-quality ratio was slightly worse: better notes, reliably slower. Set against the cost of maintaining a tuned model through the next base-model version, which is the concern promoting a fine-tuned model deals with, the product owner deferred it and asked for the comparison to be re-run after the next base release. That decision took eight minutes and left a screenshot attached to the meeting note. Not one line of it required an engineer to scroll back through a channel.
What’s worth remembering
- An evaluation result that lives in a chat thread has been produced but not delivered; naming the reader and the decision comes before choosing a visualisation tool.
- Evaluation scores arrive every few days and invocation metrics arrive every minute, so putting them on one panel makes the sparse series look broken and its averages meaningless.
- Model evaluation output is data at rest in S3, which makes the reporting choice a query choice: choose the output prefix, register it in the Glue Data Catalog, and query it with Athena.
- Quick Sight over Athena is the surface that can join scores to tagged cost and to business outcomes, which is what cost-performance analysis needs: token efficiency, the latency-to-quality ratio, and the outcome the feature exists to move.
- Bedrock emits no EventBridge event for an evaluation job, so trigger off the results landing in S3, and keep a single headline score in CloudWatch because a Quick Sight threshold alert waits for the next refresh and arrives as an email.
- Automated reporting mechanisms beat a better chart: an EventBridge schedule, a Lambda running saved queries, and an Amazon SNS topic mean the trend arrives without anyone remembering to ask for it.