The situation
A team runs a customer-facing summarisation feature on Amazon Bedrock: an inbound message thread goes in, a short summary comes back, and an agent reads it before replying. It has been on one Claude model and one hand-written prompt for months. Two changes are queued. Someone has rewritten the prompt to be tighter and, on a spreadsheet of fifty saved threads, its summaries look clearly better. Separately, a newer Claude model has landed on Bedrock that benchmarks well and costs less per thousand tokens, and finance would like the saving.
The current release process is that whoever edits the prompt commits it, it ships, and regressions surface days later as agent complaints (“the summaries have gone vague”) with no way to prove which change caused it or to get back to the old behaviour quickly. The fifty-thread spreadsheet is the only evidence anyone has, and it was assembled by the same person who wrote the new prompt.
Both changes might be improvements. The question underneath both is the same: how do you find out on real traffic whether a variant is actually better, without betting the whole live workload on a hunch, and how do you get back to safety fast if it is not?
What actually matters
The first thing to separate is what an offline score can and cannot tell you. Running the new prompt over a fixed set of saved threads, whether you grade the outputs by hand, by rubric, or with a model acting as judge, is cheap, repeatable, and safe, and it is the right first gate. What it cannot do is represent live traffic. The saved set is small, it was chosen by a human with a view, and it does not contain the weird inputs next Tuesday will bring. A variant that wins on fifty curated threads can still lose on the long tail of real messages. Offline evaluation earns a variant the right to be tried on real traffic; it does not settle the question.
The second is that a fair comparison holds everything constant except the one thing under test. If you change the prompt and the model at the same time and quality moves, you cannot attribute the move. Same inputs, same downstream handling, same measurement, one variable. That is why the two queued changes are two separate experiments, not one. It is also why the traffic each variant sees has to be comparable: route by a stable hash of something neutral like a request id, so the split is random with respect to the input and not, say, all the long threads landing on one side.
The third is the risk gradient, and it decides the whole shape of the rollout. A change you are unsure about does not go straight to a share of live users. It goes first to shadow: mirror a copy of live traffic to the new variant, throw its output away rather than showing it to anyone, and compare the two responses offline. Shadow testing exposes the variant to real inputs at real volume with zero user-facing risk, which is exactly what the fifty-thread set could not do. Only once shadow looks good does a small live split make sense, and only then a gradual ramp. The more a bad output would cost a user, the more of that ladder you climb before going live.
The fourth is that quality is not the only axis, and measuring it alone hides regressions. A variant can write better summaries and also be slower and dearer, or cheaper and faster and slightly worse. You measure three things together on every variant: quality, however you can proxy it (a model-as-judge score, a sample sent to human review, or real user feedback like the agent marking a summary useful or not); latency per request; and cost per request, which for the model swap is the entire point of the exercise. A decision that looks at quality without latency and cost is half a decision.
The fifth is that you cannot tell a real difference from noise without enough samples. LLM outputs vary, and on a handful of requests a worse variant will sometimes look better by luck. Before you read a live split as a result, it needs enough traffic that the gap between A and B is bigger than the run-to-run wobble. Small early splits are for catching disasters fast, not for declaring a narrow winner; a two-percent quality edge needs far more traffic to trust than a feature that fell over outright.
And the cross-cutting one: you can only compare and roll back cleanly if each variant is a named, versioned thing. A prompt pasted inline in code has no version to route to and no version to revert to. Bedrock prompt management holds prompts as assets with numbered versions you can point traffic at; a model is identified by its model id or Inference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. id. When both the prompt and the model under test are addressable by an identifier, the experiment is “send this share of traffic to version 4 on the new model id” and the rollback is “send it all back to version 3 on the old one”, both of them configuration changes rather than code deploys.
What we’ll filter on
- Risk tolerance, how much would a bad output cost a user, which sets how far up the shadow-then-split-then-ramp ladder you go before live exposure?
- What you measure, quality proxy plus latency plus cost per request, all three on every variant?
- How you route, can you split traffic randomly and hold everything else constant, and dial the share up and down?
- How you decide, do you have enough samples for the gap to beat the noise, and a clear threshold to promote or kill?
- Rollback speed, is reverting to the previous variant a configuration change measured in seconds, not a redeploy?
- Variant addressability, are the prompt and model each a versioned, identifiable asset you can name in the routing rule?
The rollout landscape
Offline evaluation on a fixed set. Run each variant over a saved dataset and grade the outputs, by rubric, by human, or with a model-as-judge. On Bedrock this is what model evaluation jobs are for: point an automatic or human evaluation, including a model-as-judge option, at a dataset in Amazon S3 and get comparable scores per variant. Cheap, safe, repeatable, and the natural first gate. Its ceiling is that the set is fixed and curated, so it certifies plausibility, not live superiority.
Shadow (mirror) testing. Duplicate live requests to the candidate variant and discard its responses; users only ever see the current one. Log both outputs and compare them offline, often with the same judge you used on the fixed set. This gives you real inputs at real volume with no user-facing risk, and it is the low-risk first step for any change you are unsure about. The cost is that you pay for the shadow inferences and you build a little plumbing to fan out and log, and because nobody sees the output you cannot measure genuine user feedback yet, only judged quality, latency, and cost.
Canary / live split. Route a small share of live traffic, say one or five percent, to the new variant and serve its output for real, keeping the rest on the incumbent. Now you can measure real user feedback alongside the judged score, and catch failures that only show when the output is actually used downstream. The share is a dial you raise as confidence grows and drop to zero to roll back. The exposure is real, so this comes after shadow for anything risky, and the split must be random with respect to the input.
Gradual ramp. Once a canary holds up, increase its share in steps, five to twenty-five to fifty to a hundred, watching the three metrics at each stop and pausing or reversing if any degrades. This limits the blast radius of a regression that only appears at scale and gives real feedback time to accumulate. It is slower than flipping straight to a hundred percent, which is the entire point.
The routing mechanism. Something has to decide, per request, which variant serves it, and let you change the weighting without a code deploy. A feature-flag or dynamic-configuration service such as AWS AppConfig holds the split percentage and the variant identifiers as configuration your application reads at request time, so dialling the canary from one percent to twenty-five, or back to zero, is a config change that propagates in seconds. Application-side weighted routing in your own service does the same job in code you control. The property that matters is a fast, deploy-free way to change and revert the weights.
Variant management underneath. Prompt variants live in Bedrock prompt management as numbered versions you route between; model variants are their model ids or inference profile ids. Keeping each variant addressable by an identifier is what makes the split rule and the rollback rule simple, and what lets you reproduce exactly which prompt version ran on which model when you read the results back.
Side by side
| Approach | User-facing risk | Real user feedback | Catches long-tail inputs | Speed to signal | Best first for |
|---|---|---|---|---|---|
| Offline eval on fixed set | None | ✗ | ✗ | Fast | Certifying a variant is plausible |
| Shadow / mirror | None | ✗ | ✓ | Medium | An unsure change, before any live exposure |
| Canary / live split | Low (small share) | ✓ | ✓ | Medium | First real exposure after shadow passes |
| Gradual ramp | Grows with share | ✓ | ✓ | Slow (by design) | Limiting blast radius to full rollout |
| Full cutover | High | ✓ | ✓ | Immediate | Only a change already proven by the above |
Reading it against the two queued changes: the prompt rewrite goes offline first on a bigger set than fifty threads, then shadow, then a small canary, then a ramp. The model swap follows the same ladder but leans hardest on measuring latency and cost per request, because the saving is the reason for the change and a cheaper model that summarises slightly worse is a trade to make on purpose, not by accident.
The picks in depth
The prompt rewrite is the lower-stakes change, but it still does not skip the ladder, because the only evidence for it is a spreadsheet its own author built. Register the new wording as a version in prompt management so it has a number to route to, then run both versions offline over a dataset far larger and less hand-picked than the original fifty, scored by a model-as-judge with a written rubric and a sample pulled for human review to keep the judge honest. If the new version wins there, shadow it against live traffic and compare the paired outputs; live threads will include shapes the saved set never had. Only then serve it to a small canary where agents can mark summaries useful or vague, and ramp from there. At every stage the incumbent version is the control, and rollback is pointing the weight back at the old version number.
The model swap is where measuring all three axes together does the real work. A newer, cheaper model id is attractive precisely because of cost, so the experiment has to weigh the per-request saving against any drop in judged quality and any change in latency, not just confirm the summaries are still readable. Hold the prompt constant, the exact same version, and vary only the model id or inference profile so the comparison is clean; running the model swap and the prompt rewrite as one change would leave you unable to say which one moved the numbers. Shadow is especially valuable here because it prices the new model on real traffic before a single user sees it: you get judged quality, real latency, and real cost per request at volume with no exposure. If the saving is real and quality holds within tolerance, canary and ramp; if quality slips more than the saving justifies, the change dies at shadow having cost only some mirrored inferences.
The decision rule is the part teams skip, and it is what turns a split into a result. Fix, before you start, what you are measuring, what threshold counts as a win, and roughly how much traffic you need for the gap to beat the run-to-run noise. Without that, a small early split becomes a place to stare at a dashboard and rationalise. The small canary is there to catch a variant that is plainly broken quickly; declaring a narrow quality win needs far more samples than catching a disaster does, and reading a two-percent edge off a few hundred requests is reading noise. Pair every experiment with the same rollback move regardless of outcome: the weight is a dial, zero sends all traffic home to the known-good variant, and that revert is a configuration change, not a redeploy.
A worked example: shipping the cheaper model
Start state: prompt version 3 on the old model id, serving a hundred percent of traffic, judged quality averaging well and agents mostly content. The goal is to move to the new, cheaper model id if it holds quality.
First, keep prompt version 3 fixed and stand the new model id up as the candidate; the only variable is the model. Run both offline over a few hundred saved threads with a model-as-judge and a human-reviewed sample, and record judged quality, latency, and cost per request for each. The new model comes out slightly lower on quality, meaningfully lower on cost, and a touch faster. Plausible, not yet proven.
Next, shadow. Mirror live requests to the new model, discard its summaries, and log both sides. Over a few days of real traffic the judged-quality gap holds at about a point, the cost saving holds, and, importantly, a cluster of long multi-party threads that never appeared in the saved set turns up, where the new model truncates more aggressively. That is exactly the long-tail signal the fixed set could not have shown, and it is caught with zero user exposure. Suppose the truncation is within tolerance for the agents’ use; the change survives shadow.
Then canary. Put five percent of live traffic on the new model via the split held in configuration, serve its output for real, and watch judged quality, latency, cost, and the agents’ useful-or-vague marks. The marks track the shadow finding: slightly terser, still useful. With enough canary traffic for the small quality gap to be real rather than noise, and the cost saving confirmed on live volume, ramp: five, twenty-five, fifty, a hundred, pausing at each step to check the three metrics. At any step, if judged quality or the agents’ feedback drops below the threshold set at the start, the weight goes to zero and every request is back on the old model id in seconds, no deploy. The change ships as a deliberate quality-for-cost trade you measured, not one you discovered from complaints.
What’s worth remembering
- Offline evaluation on a fixed set certifies that a variant is plausible; only live traffic tells you it is actually better, because the saved set is small, curated, and missing next week’s long tail.
- Change one thing at a time; a prompt rewrite and a model swap are two experiments, and running them together leaves you unable to attribute any movement in the numbers.
- Match the rollout to the risk: shadow first for anything you are unsure about, then a small live canary, then a gradual ramp, and only ever a full cutover for a change already proven.
- Shadow testing mirrors real traffic to the candidate and discards its output, giving you real inputs at real volume with zero user-facing risk, which is what a fixed set cannot do.
- Measure quality, latency, and cost per request together on every variant; a decision that weighs quality alone hides the regression that lives in the other two.
- Split traffic randomly with respect to the input, by hashing something neutral like a request id, so the two variants see comparable work.
- You need enough samples for the A-to-B gap to beat run-to-run noise; a small early split catches disasters fast but cannot settle a narrow quality win.
- Fix the metric, the win threshold, and the rough sample size before you start, or a live split becomes a dashboard to rationalise rather than a decision to make.
- Keep each variant addressable: prompts as numbered versions in Bedrock prompt management, models as their model ids or inference profile ids, so routing and reverting name a specific asset.
- Hold the rollout weight in feature-flag or dynamic configuration such as AWS AppConfig so dialling the canary up, or back to zero, is a configuration change that lands in seconds rather than a redeploy.