The situation
A freight company runs a customer-facing knowledge assistant. It sits behind Amazon API Gateway and a Lambda, retrieves from an Amazon Bedrock knowledge base built over the company’s tariff schedules, customs guidance, and delivery-window policies, and answers questions with a short paragraph and a list of source citations. It handles a few thousand conversations a day.
On the Tuesday, the release pipeline ran its usual gates. The Golden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions. of 600 questions scored above threshold, the guardrail suite passed, the smoke test came back green, and the change went out at lunchtime.
On the Friday, a support team lead raised a ticket. Answers had gone vague. Where the assistant used to say “surcharges apply on lanes into Zone 4 between 1 June and 31 August, see tariff 12.3”, it was now saying “surcharges may apply on some lanes during peak periods, please check with your account manager”. Citations were still present, but often pointed at a general overview document rather than the specific clause. Roughly one answer in six had drifted this way, and the drift had been going on since Wednesday morning.
Nothing had alarmed. Latency at p95 was flat. The 5xx rate was zero. Token spend was within four percent of the previous week, and the CloudWatch dashboard was a wall of green. Two changes landed on the Wednesday that nobody had connected to it. The nightly knowledge base sync had reingested a source repository that was restructured the day before, and the inference profile the application calls had begun resolving to a newer build of the same foundation model. Neither of those is a deploy. Neither shows up in the pipeline. Both can change what the assistant says.
What actually matters
The three signals the platform hands over for nothing are latency, error rate, and cost, and none of them describes whether an answer is still correct. A degraded answer is an HTTP 200 with an ordinary duration and an unremarkable token count. Everything the infrastructure can see about it looks exactly like a good answer, so a quiet dashboard is not evidence of health; it is evidence that the wrong things were measured. Once you accept that, quality becomes a signal you have to construct rather than one you subscribe to, and constructing it means deciding what a check is allowed to assert.
That decision is harder than it is for ordinary software, because a generative output cannot be asserted byte for byte. Send the same prompt twice and the wording moves; that variation is the model working as configured, not a fault. A check that compares strings fails constantly and gets muted within a week, which is worse than no check. So the assertions have to sit on properties that survive legitimate rewording and break when the substance changes. The response parses against a schema. The citation list is non-empty and resolves to real documents. Every claim in the answer is supported by the retrieved context, and the answer stays within a similarity band of a reference answer recorded when the same input was known to be answered well. AI-specific output validation means those checks. An HTTP smoke test that asserts a 200 and a non-empty body would have passed happily through all three days of this.
The changes that break quality also do not respect the release calendar. In this scenario nobody deployed anything on the Wednesday, and yet two of the three inputs to every answer moved: the retrieval corpus and the model build. Add prompt template promotion, an embedding model upgrade, a guardrail policy edit, and an upstream author rewriting a source document, and you have a list of quality-affecting events that a pipeline gate cannot see because the pipeline was not running. Deployment validation that only fires when a deploy fires misses the whole class. The check has to run on a clock, against production, using the same front door a customer uses, so it exercises the live retrieval path and the live model rather than a version of them frozen at build time.
The last property is tolerance. A per-answer score bounces around because sampling bounces around, so alarming on a single low score generates noise until somebody deletes the alarm. What is worth alarming on is the middle of the distribution moving: a rolling window of replayed cases, a band established while the system was known good, and a page when the window leaves the band. That framing also sets the cost ceiling, because it says how many cases you have to replay per window and therefore how many tokens a day the checking costs. Detection lag, token cost, and false-alarm rate are one dial, not three.
What we’ll filter on
- Detection lag: how long between the quality changing and somebody being told, measured in minutes and hours rather than days.
- Reach: does the check drive the live endpoint and its real dependencies, including auth, retrieval, and whichever model build the inference profile currently resolves to?
- Assertion power: can it only see HTTP shape and timing, or can it inspect the content of the answer against something recorded?
- Cost per day: tokens for the replayed calls, tokens for any judge, and the operational cost of maintaining the reference material.
- Tolerance for legitimate variation: does it stay quiet when the model rewords an answer, and speak up when the substance moves?
The landscape
Amazon CloudWatch Synthetics. A canary is a script that CloudWatch runs on a schedule from outside your application, as often as once a minute, driving the endpoint the way a client would. An API canary makes the HTTP calls directly; a browser canary drives the user interface with Puppeteer or Selenium and can capture screenshots and a HAR file to Amazon S3. Either shape gives you scripted synthetic user workflows: log in, ask the three questions a real user asks first, follow up on the answer, check the citation link resolves. The script’s assertions are ordinary code, so a canary can validate a JSON schema, require a non-empty citation array, reject a refusal string, and enforce a length floor. It publishes SuccessPercent and Duration as CloudWatch metrics, which alarm like any other metric. Scoring is what a canary is not built for. Running a judge model inside a script with a short timeout, every five minutes, is expensive and fragile, and one canary covers a handful of canonical journeys rather than a corpus.
A scheduled golden-set replay. An Amazon EventBridge schedule starts an AWS Step Functions state machine, which maps a slice of the golden set over the production endpoint and scores each answer. The plumbing has the same shape as the nightly run that gives a release gate its incumbent score, and the target is the difference that matters: that run drives staging so a candidate release has something to be compared against, this one drives the endpoint customers are using, so it sees the corpus and the model build that are actually answering questions rather than the ones staging was pinned to at build time. Scoring is where the automated quality checks live: a faithfulness judgement of each answer against the context that was actually retrieved for it, and a distance measurement between today’s answer and the recorded reference for the same input. Results go to a custom CloudWatch metric per check, which alarms and draws on the same dashboard as the operational signals. This gives real breadth and real assertion power, at the cost of the tokens it burns and of a reference set somebody has to maintain. Its replayed traffic also has to be tagged and excluded from usage analytics, or the synthetic calls end up in the numbers finance reads.
Shadow traffic scored by a judge. Mirror a sample of live requests to a second path, or read them back from Bedrock model invocation logs, and score the answers. The distribution is real, so it catches degradation on questions nobody put in the golden set. The weakness is that a real question has no recorded correct answer, so a judge can rate an answer on a rubric but cannot tell you it moved, and the score has no baseline other than yesterday’s score. Cost scales with traffic rather than with a slice you chose, and duplicating customer questions into a scoring pipeline is a data-handling decision on its own.
Waiting for the feedback loop. Thumbs, complaint tickets, and account managers. Nothing to build, and it does surface problems the machinery misses, which is why it stays switched on. As a detector it lags by days, samples only the users annoyed enough to say something, and delivers its finding after a customer has already been given a bad answer. This is the option the freight company was running, and Friday is what it returns.
Evaluation
Side by side
| Option | Detection lag | Drives the live endpoint | Asserts on answer content | Cost per day | Tolerates rewording |
|---|---|---|---|---|---|
| Amazon CloudWatch Synthetics canary | Minutes | ✓ | Partly (schema, citations, refusals) | Cents | ✓ |
| Scheduled golden-set replay | Under an hour | ✓ | ✓ (faithfulness, distance to reference) | Tokens for the slice plus the judge | ✓ (banded) |
| Shadow traffic to a judge | Under an hour | ✓ (mirrored) | Partly (no reference to compare to) | Scales with live traffic | ✓ |
| The user feedback loop | Days | ✓ | ✓ (a human read it) | Nothing to run, plenty to clean up | ✓ |
The two columns that separate the options are assertion power and lag, and no single row wins both cheaply. The canary is fast and almost free and can only see the shape of a response. The replay can see the substance and costs tokens every time it runs. Shadow traffic sees the real distribution and has nothing to compare it against. Reading the table as a ranking gives the wrong answer; reading it as two different questions gives the right one.
Which check catches which change
The diagram splits on a distinction that decides the whole design. “Is the workflow alive” is a cheap question with a fast answer, so ask it often. “Are the answers still right” needs a reference, a judge, and a window before it means anything, so ask it hourly and accept the lag. Trying to make one mechanism answer both produces either a canary too slow and expensive to run, or a replay too coarse to notice that the endpoint has been returning a 502 for twenty minutes.
The solution
Run both, on separate schedules, alarming into separate places.
The canary
An Amazon CloudWatch Synthetics canary, on a five-minute schedule, driving the production endpoint through the same front door a customer uses. Script it as three or four synthetic user workflows rather than a single ping: the top question by volume, a follow-up in the same conversation that depends on retained context, and a question whose correct answer is a refusal, so the guardrail path is exercised too.
Assertions are what makes it more than a ping. Per request, check the status code and the end-to-end duration, then parse the body against the response schema. Require the answer field to be present and above a length floor. Require the citations array to be non-empty, with every citation identifier resolving to a document that still exists. Reject the model’s standard “I don’t have enough information” wording on the questions that should be answerable, and require it on the question that should not be. Every one of those holds regardless of how the model words the answer, which is what a check has to do to survive in production.
Give the canary its own IAM role and its own tenant identifier so its calls are tagged and can be excluded from usage analytics, cost attribution, and any feedback-derived training data. Send artefacts to S3 with a short lifecycle rule, since a failing browser canary’s screenshot is often the fastest route to the cause. Alarm on SuccessPercent dropping below 100 across two consecutive runs, and route that alarm to whoever is on call, because it means the feature is broken rather than blunted.
The replay
An EventBridge schedule every hour, plus an extra rule that fires on the completion of a knowledge base sync so the replay runs immediately after the corpus moves rather than up to an hour later. Each run takes a rotating stratified slice of the golden set, sixty cases out of six hundred, so a full pass completes daily and every run covers each question category. A Step Functions state machine maps the slice over the production endpoint, with a concurrency limit that keeps the replay from competing with real traffic, and writes raw answers plus the retrieved context to S3.
Scoring runs in two passes, cheap first. The cheap pass is deterministic: schema validity, citation presence, and the embedding distance between today’s answer and the recorded reference answer for that input. The expensive pass invokes a judge model, and only on the cases the cheap pass flagged plus a fixed random sample of the rest, which keeps token cost roughly flat as the golden set grows. Two AI-specific output validation measures come out of it:
- Hallucination rates. Judge each answer against the context that was actually retrieved for it, claim by claim, and publish the proportion of replayed answers carrying at least one unsupported claim. This is faithfulness against the retrieved context, not against the world, which is what makes it computable on a schedule.
- Semantic drift. Embed today’s answer and the recorded reference for the same input with the same embedding model, and take the cosine distance. Publish the rolling mean over the last full pass. Alarm when that mean leaves a band established during a known-good window rather than when a single case crosses a line, because one case crossing a line is sampling variance and a fortnight of that teaches the team to ignore the alarm. This is the finer-grained cousin of the behaviour drift a version rollback is meant to undo: behaviour drift is the thing you notice, semantic drift is the number that says it is happening.
Response consistency is the third measure and comes almost free. Sample each replayed case three times instead of once at the production temperature, and record the spread of the pairwise distances between those samples alongside the mean. A widening spread with a stable mean is an early signal, because a model that has started answering the same question three different ways will eventually answer it wrongly for somebody.
The gotchas
The reference answers have to be captured, not reconstructed. Record the answer, the retrieved context, the prompt version, and the resolved model identifier at the moment the pipeline gate passed, and store them as a versioned artefact. Nobody can write a reference answer from memory three days after a regression started.
Pin the judge model and the embedding model, and version them explicitly. If either moves, every historical score becomes incomparable and the band has to be re-established from a fresh known-good window. A judge that quietly upgraded itself produces exactly the alarm pattern you are trying to detect, and there is no way to tell the two apart after the fact.
Record what changed on the same timeline as the metrics. Read the model identifier back from each invocation response and publish it as a dimension, emit an event when a knowledge base sync completes and when a prompt version is promoted, and put those annotations on the quality dashboard. Half the value of the drift metric is being able to line its step up against the change that caused it.
Route the two alarms differently. Canary failure pages, because the workflow is down. A drift band breach opens a ticket, blocks the next promotion, and triggers the comparison you would run against a candidate between the current and previous configuration. Paging at 3am on a rolling mean nobody can act on until morning is how a good signal gets switched off.
Worked example
The resync that dropped the clause
Wednesday, 02:14. The knowledge base sync completes after the tariff repository was restructured, and the clause-level documents that used to be chunked individually now sit inside larger overview pages. Retrieval still returns something for every question, so nothing errors.
Under the design above, the sync-completion rule fires the replay at 02:20. The cheap pass finds citation identifiers that no longer resolve for eleven of the sixty cases, and an embedding distance from reference that has moved from a mean of 0.11 to 0.29 on the tariff category specifically. The category dimension is what points at the cause: general policy questions are unchanged. The judge pass on the flagged cases reports that the answers are still faithful to what was retrieved, which rules out the model and points squarely at retrieval, so the on-call engineer is reading the index and its chunking rather than the prompt. Detection lag is six minutes instead of three days, and the tariff questions were answered badly for one overnight window rather than for most of a working week.
The canary, meanwhile, stays green through all of this, and it should. The journey completes, the answer parses, citations are present. Its only contribution is negative evidence, which is useful: the endpoint is fine, so the problem is in the content.
The model build that moved
Wednesday, 09:40. The inference profile begins resolving to a newer build of the same model, which is the ordinary behaviour of calling a profile rather than a pinned version and not a fault in itself. The new build hedges more, and pinning the version and re-running the set against both builds is the same remedy a nightly staging run would arrive at. What is worth watching is how the production replay gets there, with two unrelated changes now landed inside the same seven hours.
Three measures move in different directions, and the combination separates them. Citations still resolve and hallucination rates improve slightly, because a hedging model asserts less. Semantic drift climbs across every category rather than in the tariff category alone. Response consistency tightens rather than widens, which is what a more cautious build does to a spread. Set that against the resync from before dawn: drift in one category, citations broken, consistency untouched. Two signatures, distinguishable at a glance, and the model dimension published from the invocation response confirms the second one in seconds by showing a new identifier from 09:40. Without the per-category and per-measure breakdown, both events are one line sloping upwards and the on-call engineer is guessing which change to back out.
What’s worth remembering
- Latency, error rate, and cost say nothing about whether an answer is correct, so a green dashboard through a quality regression is the expected outcome rather than a surprise.
- Deployment validation has to run on a clock against production, because the model build, the retrieval corpus, and the source documents all move without a deploy.
- Amazon CloudWatch Synthetics runs scripted synthetic user workflows against the live endpoint every few minutes and can assert on schema, citations, and refusals, which covers whether the workflow is alive but not whether the answers are right.
- AI-specific output validation asserts on properties that survive rewording: hallucination rates measured against the retrieved context, and semantic drift measured as embedding distance from a reference answer recorded when the system was known good.
- Alarm automated quality checks on a rolling band rather than a single score, and measure response consistency by sampling each case several times, because a widening spread arrives before a moved mean.
- Capture the reference answers, prompt version, and resolved model identifier at the moment the gate passed, and pin the judge and embedding models, or every later comparison is measuring your own tooling.