Turning a Golden-Set Score Into a Deployment Gate

The situation

A support assistant runs on Amazon Bedrock over a knowledge base of delivery policies, returns rules and product data. The team did the hard part already. They have a golden setGolden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions. of 400 examples: real customer questions with accepted answers, a slice of known-hard cases, and about sixty questions the assistant is supposed to decline. They have a Bedrock evaluation job wired to it that scores answer relevance, groundedness, refusal correctness and harmful-output rate. It runs. It takes about forty minutes and costs a few dollars.

Nobody has to run it. Prompt changes merge when the author has read a dozen answers in a scratch notebook and is satisfied. Two regressions shipped that way. The first was a rewrite of the system prompt to make answers shorter, which relaxed a refusal along the way. The assistant started guessing at contract pricing it has no data for, and a customer found it three days later. The second was moving to a newer model version to cut latency, which cut latency and dropped groundedness by six points, noticed a fortnight afterwards in a run of complaints about invented delivery windows. Both would have been caught by a job the team already owns and did not run.

So the machinery exists and nothing consumes it. What the team wants is a score that can stop a release without stopping the team.

What actually matters

Start with what a run costs, because that constrains everything downstream. Four hundred examples means 400 generations, and when a judge model grades them, 400 more calls on top. That is tens of minutes of wall-clock and dollars of tokens per execution, every time. Attach it to every commit and a developer waits forty minutes to learn that a typo in a comment did not change the assistant’s behaviour. Gates like that do not survive contact with a deadline. People batch three changes into one push to amortise the wait, or they merge in the morning without reading the result. Within a fortnight the gate is a formality that adds latency and catches nothing. Set size and placement are therefore the first decision, and the same 400 examples do not belong everywhere. A small set can run constantly. A large set can run rarely. Neither can do the other’s job.

The second constraint is that the number moves on its own. Sampling temperature, judge-model variance, tie-breaks in retrieval ordering: run the identical commit twice and the aggregate score differs. Measure that before choosing any threshold. Run the set five times against an unchanged system and look at the spread, because that spread is the floor on what the gate can detect. If groundedness wanders by 1.5 points on identical input, a threshold one point under yesterday’s score fails good changes most weeks, and a threshold five points under waves through exactly the six-point drop that shipped last month. The false-block rate and the miss rate are the same dial, and the measured wobble is where it sits. What comes out of that is a rule rather than a number: a comparison against what is currently running, a band in the middle where the result is inconclusive, and something to do when a result lands in it.

The third is that a failure has to cost the right amount, and the consequence should track how much the run actually knows. Thirty examples are confident about breakage and know almost nothing about quality, so a failure there should stop a merge and no more. Four hundred examples at a promotion boundary know enough to stop a production deploy. A check on live traffic knows the most and arrives too late to prevent anything, so its consequence is a rollback. Bolt the strongest consequence onto the weakest signal and the team spends its week re-running a flaky gate; bolt the weakest consequence onto the strongest signal and there was no gate to begin with.

And the override belongs in the design rather than in the incident. Someone will need to ship a customer-facing fix at six on a Friday while the evaluation queue is backed up. If the escape hatch was never designed, it is whoever holds pipeline permissions clicking through with no record. Name the role that may override, require a written reason, expire the override at one release, and count them. A gate that gets overridden eleven times a quarter is not gating anything, and the count is what makes that visible instead of arguable. The same reasoning applies to what the run leaves behind. A bare score is unusable three months later when someone asks whether the June prompt change caused the drop. The run needs its inputs and outputs kept: which version of the golden set, which prompt and guardrail versions, which model identifier and inference configuration, which judge model and rubric, the per-example responses, and the verdict with the threshold that produced it. That record settles the argument by re-reading rather than by memory, and it is what you hand an auditor who asks how you know quality held across a model change.

What we’ll filter on

1. Wall-clock: how long from a change landing to a verdict, and is a human waiting for it?
2. Cost per run: is this affordable at the frequency the placement implies?
3. False-block rate: given the measured wobble, how often does this placement fail a change that was fine?
4. What a failure stops: a merge, a promotion, live traffic, or nothing except a notification?
5. Durable evidence: does the run leave an artefact with scores, versions and per-example outputs that outlives the execution?

The landscape

The evaluation machinery is not the open decision. Amazon Bedrock Model Evaluations takes a prompt dataset from Amazon S3 and generates responses from the model or endpoint under test. It scores them with automatic metrics, a human review workflow, or an LLM-as-a-judgeLLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. rubric, then writes the results back to S3 as job output. That job is the same job wherever it is called from. What differs is how much of the set it runs, what starts it, and what its verdict is allowed to do.

A smoke set on every pull request. Twenty to forty examples run from an AWS CodeBuild step attached to the pull request. They cover the answer shapes the assistant handles most, plus the failure modes it has actually shipped. Three to five minutes, cents per run, cheap enough that nobody notices it. What it catches is breakage: a prompt template that no longer renders, a guardrail configuration that now refuses every legitimate question, a tool schema the agent cannot parse, a refusal slice that has collapsed to zero. What it cannot catch is a two-point drift, because thirty examples cannot distinguish two points from noise. Failing here stops a merge.

The full set on a schedule. An Amazon EventBridge schedule starts an AWS Step Functions execution overnight. The state machine starts the evaluation job against the deployed staging environment, waits for it to finish, reads the scores out of S3 and records them. This is the continuous evaluation workflow, and it is doing something the pipeline gates cannot: running the same 400 examples against a moving system on a fixed cadence, producing a time series instead of a verdict. It catches the drift that no commit caused, which is the category the team has been worst at. A model version rolling forward under an alias, an index re-synced with different chunking, a retrieval library upgrade that changed tie-break ordering: none of those arrive as a pull request, so none of them meet a pull-request gate. The nightly run blocks nothing. Its output is the incumbent score that every other gate compares against, plus the trend that says whether the system is sliding.

A hard gate before the production deploy. A stage in AWS CodePipeline between staging and production starts the job over the full set, waits, and passes or fails the stage on the resulting scores. This is regression testing for model outputs in its most direct form, and it is the automated quality gate for deployments that the team is missing. The run happens because a release is trying to move, and the release does not move until the run says it may. The waiting is the awkward part, since a pipeline action has to sit through forty minutes of asynchronous work. That is why the pipeline this sits inside hands the waiting to Step Functions, which is the service happy to sit idle for an hour. A CodeBuild action starts the state machine execution and waits on it, or the state machine calls the CodePipeline API to signal the result when the evaluation job finishes, so no build script is left holding its own timeout and retry logic around a forty-minute job. Highest confidence available before exposure, and the highest wall-clock cost.

A check after the change is live. Once the release is out to a share of traffic, the material to score stops being the set and becomes the traffic. Canary testing routes a small percentage of requests to the new configuration, a sample of the real responses gets scored by a judge model or by explicit user feedback, and a CloudWatch alarm on that rate triggers the rollback. Over a longer window, A/B testing two variants against each other measures what the golden set can only approximate. That is the cost-performance analysis a swap actually turns on, meaning token efficiency and the latency-to-quality ratio at production volume. It also covers the business outcomes, deflection rate and escalation rate, that the accepted answers were always a proxy for. Nothing here prevents a bad release. It shortens the exposure and it settles the multi-model comparisons the golden set argues about.

Evaluation

Side by side

Read down the two tick columns and no row is both affordable on every commit and able to see a small regression. That is not a gap in the tooling; a set small enough to run per commit cannot resolve two points, and a set large enough to resolve two points cannot run per commit. Read across the failure column and the four placements stop four different things at four different moments. Picking one row means giving up three of those moments, which is why the answer is a ladder rather than a choice.

The solution

Run all four, with different sets and different consequences, and let each one be judged on what it is capable of knowing.

The ladder

Every pull request runs the thirty-example smoke set from a CodeBuild step against the dev deployment. It has absolute floors, set generously, because it exists to catch a system that is broken rather than a system that is slightly worse. A failure stops the merge and takes five minutes to confirm or clear.

The nightly run is the continuous evaluation workflow, and it is where the reference numbers come from. An EventBridge schedule starts a Step Functions execution at two in the morning. The state machine polls the knowledge base ingestion to completion first, so the run is not scoring a half-synced index. Then it starts the Bedrock evaluation job over all 400 examples against staging, waits, and reads the output from S3. The aggregates go into a DynamoDB table of scores by date, and a CloudWatch metric goes out per dimension. Nothing fails because of it. What it produces is the incumbent: the most recent trusted score for each metric, plus enough history to see a slide that no single run would show.

The pre-deploy gate runs the same 400 examples against staging as a CodePipeline stage, and compares its result to that incumbent. It is the only gate with the authority to stop a production promotion, and it is the one worth the forty minutes because it is the last moment before customers are involved.

After the deploy, canary traffic carries the new configuration for a defined window while a judge model scores a sample of live answers and user feedback signals accumulate. A CloudWatch alarm on the sampled score or on the thumbs-down rate triggers the rollback. Where the change is a model swap rather than a prompt edit, this window extends into a proper A/B comparison, because swapping the model behind the feature turns on token efficiency and latency at production volume as much as it turns on the golden-set score.

Thresholds as a delta, with two exceptions

Absolute floors go stale. A floor of 0.88 set eighteen months ago against a model two generations back tells you nothing about whether today’s change made things worse. Either it sits so far below current performance that it never fires, or somebody has been quietly nudging it. Express the gate as a delta instead: this run against the most recent nightly incumbent, per metric, with a band derived from the measured wobble. Groundedness that wanders 1.5 points on identical input gets a 1.5-point band, so a change scoring level or better passes, a change more than 1.5 points down fails, and anything in between is inconclusive rather than a coin flip.

Two metrics keep absolute floors on top of the delta. Harmful-output rate has an acceptable value of zero, and “no worse than last night” is not a sentence anyone wants read back to them. Refusal correctness has a floor because the sixty should-refuse examples are the slice that caused the first of the two regressions, and a change that is only slightly worse at declining is still shipping guesses about pricing. For everything else, the delta is the gate and the incumbent is the reference.

When a result lands in the band

Repeat the run. Same commit, same set version, same configuration, once more. Two runs inside the band means the change is indistinguishable from what is running, so the stage passes and the result is flagged in the release record as a near miss. A second run below the band means the first was not noise, and the stage fails. The rule goes into the state machine, decided in advance, because the alternative is deciding it at four in the afternoon with a release waiting and everybody’s judgement pointing the same direction.

The override that stays a gate

One named role may override a failed gate. The override requires a written reason recorded against the release, expires after that single promotion, and shrinks the canary: an overridden change goes out to a smaller share of traffic for longer, with the post-deploy alarm thresholds tightened. The count of overrides gets reviewed monthly alongside the score trend. Two in a quarter is a working escape hatch. Eleven means the thresholds are wrong or the set has stopped matching production, and either is a fixable problem that only shows up if somebody is counting.

What each run leaves behind

The run writes a manifest to a versioned S3 bucket, keyed by pipeline execution, and the scores alone are the least useful part of it:

run: 2026-08-20T02:14:07Z
commit: 4f19c2a
golden_set: { version: v11, examples: 400, sha256: 9c1e... }
model: { id: <model-id>, inference_profile: <arn>, temperature: 0.2, top_p: 0.9 }
prompt_version: 14
guardrail_version: 4
judge: { model: <judge-model-id>, rubric_version: 3 }
scores:
  groundedness:        { value: 0.91, incumbent: 0.92, band: 0.015, verdict: pass }
  answer_relevance:    { value: 0.89, incumbent: 0.90, band: 0.020, verdict: pass }
  refusal_correctness: { value: 0.96, floor: 0.95, verdict: pass }
  harmful_output_rate: { value: 0.00, floor: 0.00, verdict: pass }
per_example_output: s3://evals/runs/4f19c2a/responses.jsonl
verdict: pass

Per-example responses go alongside as a JSONL object, so a regression can be traced to the specific questions that moved rather than to an aggregate that dropped for unknown reasons. Lifecycle rules move the responses to cheaper storage after ninety days and keep the manifests indefinitely, because the manifests are small and they are what gets read in an audit. Three months later, “how do you know the model change did not degrade refusals” is answered by reading two files instead of by recollection.

Worked example

A prompt change that lands in the band

A commit rewrites the retrieval instructions so answers cite the policy section they came from. Smoke passes in four minutes. The pre-deploy gate runs the full set: groundedness 0.905 against an incumbent of 0.92 and a band of 0.015, which is exactly on the edge, and answer relevance level with the incumbent. Inconclusive, so the state machine repeats the run without asking anyone. The second run returns 0.918. Two results inside the band, the stage passes, the release record carries the near-miss flag and both manifests. Total cost is one extra evaluation run and forty minutes nobody was watching, which is what the rule bought instead of an argument.

A model version the nightly run catches

No commit lands for nine days. On the tenth night, the scheduled run returns groundedness at 0.86 against a fortnight sitting at 0.92, and the CloudWatch metric crosses its alarm. Nothing was deployed, so the pipeline gates had nothing to fire on. The manifest names a different model identifier from the previous night: the inference profile resolved to a new minor version under the alias. The per-example file shows the drop concentrated in the long multi-part questions, where the newer version summarises instead of quoting. The fix is pinning the version explicitly and re-running the set to confirm 0.92 returns. This is the regression the team shipped for a fortnight last time, found overnight by a run that blocks nothing.

What’s worth remembering

1. A run’s set size decides its frequency and its frequency decides its consequence: thirty examples per commit stop a merge, four hundred at a promotion boundary stop a deploy, and live traffic afterwards stops exposure.
2. Measure the score’s run-to-run wobble against an unchanged system before setting any threshold, because that spread is the floor on what the gate can detect and the source of every false block.
3. Gate on a delta against the current incumbent rather than an absolute floor, keeping absolute floors only for metrics whose acceptable value is fixed, such as harmful-output rate and refusal correctness.
4. Decide in advance what an inconclusive result does; a repeat run resolved by rule beats a judgement call made with a release waiting.
5. A continuous evaluation workflow on a schedule catches the drift that arrives without a commit, which no pull-request or pre-deploy gate can see.
6. Write the golden-set version, model and prompt versions, judge configuration and per-example outputs to durable storage with every run, and design the override to be named, recorded, expiring and counted.