The situation
A team runs a customer-facing summarisation feature on Amazon Bedrock: an inbound message thread goes in, a short summary comes back, and an agent reads it before replying. It has been on one Claude model and one hand-written prompt for months. Two changes are queued. Someone has rewritten the prompt to be tighter and, on a spreadsheet of fifty saved threads, its summaries look clearly better. Separately, a newer Claude model has landed on Bedrock that benchmarks well and costs less per thousand tokens, and finance would like the saving.
The current release process is that whoever edits the prompt commits it, it ships, and regressions surface days later as agent complaints (“the summaries have gone vague”) with no way to prove which change caused it or to get back to the old behaviour quickly. The fifty-thread spreadsheet is the only evidence anyone has, and it was assembled by the same person who wrote the new prompt.
Both changes might be improvements. Both turn on the same thing. How do you find out on real traffic whether a variant is better, without exposing the whole live workload to a hunch, and how do you get back to safety fast if it is not?
What actually matters
The first thing to separate is what an offline score can and cannot tell you. Running the new prompt over a fixed set of saved threads, whether you grade the outputs by hand, by rubric, or with a model acting as judge, is repeatable and safe, and it is the right first gate. What it cannot do is represent live traffic. The saved set is small, it was chosen by a human with a view, and it does not contain the weird inputs next week’s traffic will bring. A variant that wins on fifty curated threads can still lose on the long tail of real messages. Offline evaluation qualifies a variant for a trial on real traffic. It does not settle the matter.
The second is that a fair comparison holds everything constant except the one thing under test. If you change the prompt and the model at the same time and quality moves, you cannot attribute the move. Same inputs, same downstream handling, same measurement, one variable. That is why the two queued changes are two separate experiments, not one. It is also why the traffic each variant sees has to be comparable: route by a stable hash of something neutral like a request id, so the split is random with respect to the input and not, say, all the long threads landing on one side.
The third is the risk gradient, which sets the shape of the rollout. A change you are unsure about does not go straight to a share of live users. It goes first to shadow: mirror a copy of live traffic to the new variant, throw its output away rather than showing it to anyone, and compare the two responses offline. Shadow testing exposes the variant to real inputs at real volume with zero user-facing risk, which is exactly what the fifty-thread set could not do. Only once shadow looks good does a small live split make sense, and only then a gradual ramp. The more harm a bad output would do to a user, the more of that ladder you climb before going live.
The fourth is that quality is not the only axis, and measuring it alone misses regressions. A variant can produce better summaries and also be slower and dearer, or cheaper and faster and slightly worse. You measure three things together on every variant: quality, however you can proxy it (a judge-model score, a sample sent to human review, or real user feedback like the agent marking a summary useful or not); latency per request; and cost per request, which is the reason for the model swap. A decision that looks at quality without latency and cost is half a decision.
The fifth is that you cannot tell a real difference from noise without enough samples. LLM outputs vary, and on a handful of requests a worse variant will sometimes look better by luck. Before you read a live split as a result, it needs enough traffic that the gap between A and B is bigger than the run-to-run wobble. Small early splits are for catching disasters fast, not for declaring a narrow winner; a two-percent quality edge needs far more traffic to trust than a variant that fell over outright.
And the cross-cutting one: you can only compare and roll back cleanly if each variant is a named, versioned thing. A prompt pasted inline in code has no version to route to and no version to revert to. Prompt management in Amazon Bedrock stores prompts as versions numbered from 1, each with its own ARN. You pass that ARN as the modelId on a Converse call, in place of a model id or Inference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. id. The model and the inference parameters belong to the version too, so one identifier names the whole arm. The experiment is then “send this share of traffic to version 4”. The rollback is “send it all back to version 3”. Both are configuration changes rather than code deploys.
What we’ll filter on
- Risk tolerance, how much harm would a bad output do to a user, which sets how far up the shadow-then-split-then-ramp ladder you go before live exposure?
- What you measure, quality proxy plus latency plus cost per request on every variant, with quality broken out per cohort as well as pooled?
- How you route, can you split traffic randomly and hold everything else constant, and dial the share up and down?
- How you decide, do you have enough samples for the gap to beat the noise, and a clear threshold to promote or kill?
- Rollback speed, is reverting to the previous variant a configuration change rather than a redeploy?
- Variant addressability, is each arm a versioned identifier you can name in the routing rule and revert to?
The landscape
Offline evaluation on a fixed set. Run each variant over a saved dataset and grade the outputs, by rubric, by human, or with a judge model. Bedrock evaluations read a JSONL dataset from Amazon S3, up to 1,000 prompts per job. A programmatic job and a judge job each score one model, so every variant gets its own job and you compare the reports; a human job takes up to two inference sources and rates them side by side. Cheap, safe, repeatable, and the natural first gate. Its ceiling is that the set is fixed and curated, so it certifies plausibility, not live superiority.
Shadow (mirror) testing. Duplicate live requests to the candidate variant and discard its responses; users only ever see the current one. Log both outputs and compare them offline, often with the same judge you used on the fixed set. This gives you real inputs at real volume with no user-facing risk, and it is the low-risk first step for any change you are unsure about. You pay for the shadow inferences and build the plumbing to fan out and log. Nobody sees the output, so there is no user feedback to measure yet, only judged quality, latency, and cost.
Canary / live split. Route a small share of live traffic, say one or five percent, to the new variant and serve its output for real, keeping the rest on the incumbent. Now you can measure real user feedback alongside the judged score, and catch failures that only show when the output is actually used downstream. The share is a dial you raise as confidence grows and drop to zero to roll back. The exposure is real, so this comes after shadow for anything risky, and the split must be random with respect to the input.
Gradual ramp. Once a canary holds up, increase its share in steps, five to twenty-five to fifty to a hundred, watching the three metrics at each stop and pausing or reversing if any degrades. This limits the blast radius of a regression that only appears at scale and gives real feedback time to accumulate. It is slower than flipping straight to a hundred percent, and the slowness is what limits the damage.
The routing mechanism. Something has to select, per request, which variant serves it, and let you change the weighting without a code deploy. AWS AppConfig holds the split percentage and the variant identifiers as configuration your application reads at request time, from the local cache the AppConfig Agent keeps. Its experimentation feature, generally available since June 2026, goes further: you define treatments, allocate traffic across them, and target audiences with a rule builder. A deployment strategy controls how fast a change reaches your hosts, from AppConfig.AllAtOnce to the recommended AppConfig.Linear20PercentEvery6Minutes, and a CloudWatch alarm firing during the bake window rolls the deployment back for you. Amazon CloudWatch Evidently used to serve this role; AWS discontinued it on 17 October 2025, so it is no longer a choice. Application-side weighted routing in your own service does the same job in code you control.
Variant management underneath. An arm of a split has to point at something you can name. Prompt management holds each wording as a numbered version with its own ARN. Variants are a build-time device for comparing wordings in the prompt builder, and a saved version carries one of them, so the version is the thing you route to. Because the version fixes the model and the inference parameters as well, a model swap means a second version whose template matches the first and whose model id differs. Where the change under test is a whole pipeline, retrieve then rewrite then generate then post-process, the arm is the flow. Amazon Bedrock Flows, which was called Prompt Flows before GA, publishes that pipeline as an immutable version, and you swap arms by repointing an alias. An arm named by ARN also makes the result reproducible months later, because you can say exactly which prompt version ran on which model. The rollback is a pointer change rather than a redeploy.
Evaluation
Side by side
| Approach | User-facing risk | Real user feedback | Catches long-tail inputs | Speed to signal | Best first for |
|---|---|---|---|---|---|
| Offline eval on fixed set | None | ✗ | ✗ | Fast | Certifying a variant is plausible |
| Shadow / mirror | None | ✗ | ✓ | Medium | An unsure change, before any live exposure |
| Canary / live split | Low (small share) | ✓ | ✓ | Medium | First real exposure after shadow passes |
| Gradual ramp | Grows with share | ✓ | ✓ | Slow (by design) | Limiting blast radius to full rollout |
| Full cutover | High | ✓ | ✓ | Immediate | Only a change already proven by the above |
Reading it against the two queued changes: the prompt rewrite goes offline first on a bigger set than fifty threads, then shadow, then a small canary, then a ramp. The model swap follows the same ladder but leans hardest on measuring latency and cost per request, because the saving is the reason for the change and a cheaper model that summarises slightly worse is a trade to make on purpose, not by accident.
The solution
The prompt rewrite is the lower-stakes change, but it still does not skip the ladder, because the only evidence for it is a spreadsheet its own author built. Register the new wording as a version in Prompt management so the arm is a version ARN rather than a line in a commit. Then run both versions offline over a dataset far larger and less hand-picked than the original fifty, up to the 1,000 prompts one job takes. Score each with a judge model against a written rubric, and pull a sample for human review to check the judge. If the new version wins there, shadow it against live traffic and compare the paired outputs; live threads will include shapes the saved set never had. Only then serve it to a small canary where agents can mark summaries useful or vague, and ramp from there. At every stage the incumbent version is the control, and rollback is pointing the weight back at the old version ARN. If the rewrite touches the retrieval or post-processing steps as well as the wording, the arm is a flow version in Amazon Bedrock Flows instead, and the same move is an alias pointed back one version.
The model swap is where measuring all three axes together does the real work. A newer, cheaper model id is attractive because of the saving, so the experiment has to weigh that saving against any drop in judged quality and any change in latency. Hold the wording byte-identical and change only the model, which here means a second prompt version with the same template and a different model id. Running the model swap and the prompt rewrite as one change would leave you unable to say which one moved the numbers. Shadow is especially valuable here, because it measures the new model on real traffic before a single user sees it. Every Converse response carries a latencyMs figure and its input and output token counts, so judged quality, real latency and real cost per request all come out of the shadow run at volume with no exposure. If the saving is real and quality holds within tolerance, canary and ramp. If quality slips more than the saving justifies, the change stops at shadow, and the only spend is the mirrored inferences.
Fairness is one of the things a split can compare, and it goes in while the arms are running rather than into a review afterwards. Score each arm’s outputs per cohort as well as in aggregate, splitting traffic you are already logging by the lines your users actually fall along: language, region, account tier, thread length, whatever the summariser plausibly handles differently. An arm that lifts mean judged quality while widening the gap between cohorts is a regression, and the headline average cannot show it, because the cohort that got worse is outnumbered by the one that got better. The scores come out of the pass you are running anyway. Tag each record in the evaluation dataset with its cohort in the category field, and a single judge job reports scores per category alongside the overall number, against the same written rubric you calibrated against human labels. Publish those per-cohort scores as fairness metrics in Amazon CloudWatch, next to latency and cost per request. The fairness line then sits on the dashboard the business metrics are already on, and the same threshold rule can stop a ramp: if any cohort falls below its floor, the weight goes to zero even when the average is up.
The decision rule is the part teams skip. Fix, before you start, what you are measuring, what threshold counts as a win, and roughly how much traffic you need for the gap to beat the run-to-run noise. Without that, a small early split becomes a place to stare at a dashboard and rationalise. The small canary is there to catch a variant that is plainly broken quickly; declaring a narrow quality win needs far more samples than catching a disaster does, and reading a two-percent edge off a few hundred requests is reading noise. Pair every experiment with the same rollback move regardless of outcome: the weight is a dial, zero puts all traffic back on the known-good variant, and that revert is a configuration change, not a redeploy.
Worked example
Start state: prompt version 3, which pins the old model id, serving a hundred percent of traffic. Judged quality averages well and the agents are mostly content. The goal is to move to the new, cheaper model id if it holds quality.
First, publish version 4 with the same template and the new model id, so the only variable is the model. Run both offline over a few hundred saved threads with a judge model and a human-reviewed sample, and record judged quality, latency, and cost per request for each. The new model comes out slightly lower on quality, meaningfully lower on cost, and a touch faster. Plausible, not yet proven.
Next, shadow. Mirror live requests to version 4, discard its summaries, and log both sides. Stamp each call with its arm in the requestMetadata map, which takes up to sixteen key-value pairs, so the invocation logs separate cleanly afterwards. Over a few days of real traffic the judged-quality gap holds at about a point and the cost saving holds. A cluster of long multi-party threads that never appeared in the saved set also turns up, and on those the new model truncates more aggressively. That is exactly the long-tail signal the fixed set could not have shown, and it is caught with zero user exposure. Suppose the truncation is within tolerance for the agents’ use; the change survives shadow.
Then canary. Put five percent of live traffic on version 4 via the split held in configuration, serve its output for real, and watch judged quality, latency, cost, and the agents’ useful-or-vague marks. The marks track the shadow finding: slightly terser, still useful. With enough canary traffic for the small quality gap to be real rather than noise, and the cost saving confirmed on live volume, ramp: five, twenty-five, fifty, a hundred, pausing at each step to check the three metrics. At any step, if judged quality or the agents’ feedback drops below the threshold set at the start, the weight goes to zero and every request is back on version 3 as soon as the configuration deployment reaches the hosts. No code deploy is involved. The change ships as a deliberate quality-for-cost trade you measured, not one you discovered from complaints.
What’s worth remembering
- Offline evaluation on a fixed set certifies that a variant is plausible; only live traffic tells you it is actually better, because the saved set is small, curated, and missing next week’s long tail.
- Change one thing at a time; a prompt rewrite and a model swap are two experiments, and running them together leaves you unable to attribute any movement in the numbers.
- Match the rollout to the risk: shadow first for anything you are unsure about, then a small live canary, then a gradual ramp, and only ever a full cutover for a change already proven.
- Measure quality, latency, and cost per request together on every variant; a decision that weighs quality alone misses the regression sitting in the other two.
- Fix the metric, the win threshold, and the rough sample size before you start, or a live split becomes a dashboard to rationalise rather than a decision to make.
- Score each arm per cohort as well as in aggregate, and put the per-cohort scores on the same dashboard as latency and cost; an arm that lifts the mean while widening the gap between cohorts is a regression the headline number cannot show you.