Exam Room · Advanced Generative AI Developer

Surviving a Model Deprecation on Bedrock

· 28 min read

Generative AI Development · part of The Exam Room

The situation

A subscription team runs three LLM features on Amazon Bedrock: a ticket classifier, a reply drafter, and a data-extraction job that turns free-text emails into records. All three call one foundation model, named by an explicit version id in a config file set about eighteen months ago. It has been reliable ever since, which is why nobody has touched it.

Then a notice lands. The model version they depend on is moving to Legacy, with an end-of-life date about six months out. After that date the model is removed from every Region and calls to it fail. A newer version of the same family is available, along with a couple of newer families, but none is a drop-in guarantee. The extraction job is sensitive to output format, and one of the three features runs on a fine-tuned model with a Provisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. commitment attached.

The team has two bad instincts to resist. One is to wait until the deadline forces a panicked swap. The other is to flip everything to the newest model this afternoon. Neither is a plan. What they need is a way off a retiring version without breaking production, and an application shape that makes the next deprecation routine.

What actually matters

Model versions are a stability contract, and deprecation is that contract expiring. Pinning an explicit version id is the right default, because the model behind a feature then does not change between deployments. What comes with it is a migration on AWS’s calendar rather than yours. The opposite posture, always calling the newest model in a family, avoids the scheduled migration and exposes you to silent behaviour change. A prompt that worked yesterday starts formatting its answer differently, nothing errors, and you find out from a downstream parser or a customer.

The lifecycle has more structure than that, and the details change what a runway is worth. Bedrock reports a model as ACTIVE or LEGACY in the modelLifecycle field of GetFoundationModel, alongside legacyTime, publicExtendedAccessTime and endOfLifeTime. Models launched before 7 September 2026 sit in Legacy for at least six months. For models launched after that date, the model card names the legacy period, and it is either six months or 45 days. Read the card before you pin. Forty-five days is not a quarter of planning.

Three Legacy restrictions land on a migration in progress. New Provisioned Throughput cannot be created for a Legacy model, and new fine-tuning jobs cannot be started against it. After a minimum of three months in Legacy, a model can enter public extended access, where the provider may raise the price. And an account that has not called a Legacy model for 15 days can lose access to it, so a rollback nobody exercises may not be there when it is reached for.

The second concern is how tightly the application is wired to one model’s request shape. Hand-built payloads against a specific model mean that changing the model id means rewriting request construction, and that friction turns a migration into a project. Converse gives one request and response shape across every Bedrock model that supports messages, covering tool use and guardrails, with ConverseStream for streaming. Model-specific inference parameters still go through a per-model structure, and two models still answer the same prompt differently. What disappears is the mechanical rewrite.

The third is proving the successor behaves before you trust it. A model swap is a behaviour change even within one family, and the honest check is a saved Golden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions. scored against both. Bedrock evaluations run programmatic jobs on your own prompt dataset, and judge-model jobs where a second LLM scores each response and explains the score. Those jobs accept foundation models, customised models, imported models and Provisioned Throughput models as targets. A home-grown replay harness does the same work. What matters is that the comparison runs before cutover.

The fourth is the cutover mechanism. A candidate that passes the eval set can still behave differently under live traffic, so the change has to be reversible in seconds rather than redeployed over minutes. A config value or feature flag selecting the model id, rolled out to a slice of traffic first, turns a bad successor into a flag flip. The rollback only exists while the old version is still invokable, which is a second reason to start well before the end-of-life date.

The fifth cuts across the others. Prompts and few-shot examples are tuned to one model, not universal. A successor may need the instruction reworded, the examples swapped, or the output contract restated. Plan that retuning into the migration rather than assuming the prompt library travels unchanged.

Custom models are the heavier case, and the two kinds differ. A model fine-tuned on Bedrock is trained against a specific base version, so a deprecated base can mean re-tuning against the successor base rather than repointing an id. Serving it usually means Provisioned Throughput, sold in Model unitThe billing block Provisioned Throughput is sold in – one unit delivers a fixed tokens-per-minute rate for a specific model. by the hour, with an optional one- or six-month term at a lower rate. On-demand serving for a customised model exists as a custom model deployment, but only in us-east-1 and us-west-2, only on a short list of base models, and only for models customised on or after 16 July 2025. Custom Model Import works differently again: your own weights from S3, served on demand, with no Bedrock base version underneath to deprecate.

What we’ll filter on

  1. Behaviour stability, does the feature need identical output over time, or can it absorb drift?
  2. Migration lead time, how long is the legacy period on the model card, and how much work is the move?
  3. Request-shape coupling, how tightly is the app wired to one model’s native payload?
  4. Regression detection, can we score the successor before it takes traffic?
  5. Cutover and rollback safety, can we switch, canary, and revert without a redeploy?
  6. Custom-model weight, is a fine-tuned or imported model in the path, and what serves it?

The landscape

Explicit version pinning. Bedrock model ids carry a version suffix, in the shape anthropic.claude-sonnet-4-20250514-v1:0, an id that is itself Legacy now with an October 2026 end-of-life date. Pinning one keeps a feature calling exactly that model until you change it, which is why production behaviour holds steady between deploys. The trade is a scheduled migration you own.

Floating to the newest version. Always reaching for the latest model in a family avoids the forced migration and lets the provider change your output without warning. Quality often improves. Format, tone and edge-case handling can shift with no error raised. Reasonable for human-read features, risky for anything a machine parses.

Model lifecycle status. A model is ACTIVE or LEGACY, and after its end-of-life date it is gone from every Region. AWS announces the transition by email, on the AWS Health Dashboard, in the Bedrock console, and through the API, so the notice arrives with time left to act on it. The model card carries an “EOL no sooner than” date and the length of the legacy period; the API carries the exact timestamps once they are set.

Inference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code.. Many calls go through a cross-Region inference profile rather than a bare model id. A profile defines a model and the Regions requests can route to, so it inherits that model’s lifecycle. Profiles do not support Provisioned Throughput, so a fine-tuned deployment is called directly instead.

The Converse API. One request and response shape across models that support messages, so the model id becomes a parameter. It removes the mechanical work of switching and none of the behavioural risk, which is why an eval set still matters. Per-model Invoke payloads tie each feature to one model’s quirks and make every migration a rewrite.

Bedrock evaluations. A regression check against a saved dataset turns a swap into a decision. Programmatic jobs score a candidate on your own prompt dataset. Judge-model jobs have a second LLM score and explain each response, and human evaluation jobs bring reviewers in. A custom replay harness does the same work for teams that already hold golden data.

Custom and fine-tuned models. A fine-tuned model is bound to the base version it was trained on. Once that base goes Legacy you cannot start new fine-tuning jobs against it or create new Provisioned Throughput for it, though existing throughput and existing on-demand deployments keep working. An imported model is your own weights and carries no base version at all.

Evaluation

Side by side

Option Stable output Notice needed Low coupling Scores the successor Instant rollback Cost shape
Pin explicit version ✓ Legacy period ✗ ✗ ✗ On-demand tokens
Float to newest ✗ None ✗ ✗ ✗ On-demand tokens
Converse for the call n/a n/a ✓ ✗ ✗ No charge of its own
Saved eval set n/a n/a ✗ ✓ ✗ Evaluation job
Flagged canary cutover n/a n/a ✗ ✗ ✓ Negligible
Fine-tuned on Provisioned Throughput ✓ Re-tune, so longer ✗ ✗ Dual-run only Hourly per model unit

The bottom five rows compose with each other rather than competing. Reading the table against the three features: the classifier and drafter want a pinned version reached through Converse, an eval set, and a flag-controlled cutover. The fine-tuned extraction path wants all of that plus a re-tune against the successor base and a planned overlap of old and new Provisioned Throughput. Floating to the newest model is off the table for extraction the moment a parser depends on its output shape.

The solution

Five steps, and the order matters. Pin an explicit version in config rather than inline, so the id is one value to change and roll out like any other setting. Monitor the notices on all four channels: email, Health Dashboard, console, and a scheduled GetFoundationModel call that reads modelLifecycle and alerts on LEGACY. That last one catches the case where the email reached an address nobody reads. Test the successor against the saved eval set on the same inputs as the incumbent, treating a format change or a quality drop as a blocker to investigate. Cut over behind a flag, to a canary slice first, watching quality and error signals before widening. Keep a rollback by leaving the old version invokable until the successor has held on real traffic.

That last step needs care, because a Legacy model can be withdrawn from an account after 15 days without a call. Send it a small, regular slice of traffic, or a synthetic call on a schedule, for as long as you want the option. Watch the rate too. Three months into the legacy period the model can enter public extended access, where the provider may raise the price. Existing Provisioned Throughput keeps its agreed pricing through that change.

Two details ride alongside the five steps. Prompts and few-shot examples are tuned to the incumbent, so expect to retune for the successor, and the eval set is what tells you whether the old prompt still holds. And Converse is what keeps the id swap from becoming a rewrite: if the app already speaks Converse, changing the model behind a feature is a config change plus a validation pass, not a change to request construction.

The fine-tuned model deserves its own timeline. It is trained against a specific base version, so a deprecated base can mean a re-tune: a training job to schedule, a new artefact to evaluate, and new serving capacity to stand up. Start that the day the notice lands. Once the base is Legacy you cannot launch a new fine-tuning job against it, and you cannot create new Provisioned Throughput for it either. Existing throughput keeps running, so plan an overlap where the old commitment and the new one both exist, evaluate and canary the re-tuned model, then delete the old Provisioned Throughput. Billing runs until you delete it, and a committed term cannot be deleted early.

The Bedrock model migration flow A flow from pinned version through deprecation notice, evaluation against a saved set, canary cutover behind a flag, and full rollout, with a rollback path back to the pinned version. 1. Pin version explicit id in config, reached via Converse 2. Monitor notices Health Dashboard; legacy + EOL date 3. Test successor run saved eval set; retune prompt if needed Passes? quality + format hold 4. Cut over on flag canary a slice, watch live signals Full rollout retire old version 5. Rollback flip flag to pinned version yes no: fix or hold bad on live traffic: revert while old version still invokable

Worked example

The classifier calls a model through a version id in a config value, over Converse, and has run untouched for eighteen months. A notice marks that version Legacy with an end-of-life date six months out and names a successor in the same family.

Day one, not month five. The team already holds a saved eval set for the classifier: a few hundred tickets with known-correct labels, taken from real traffic and frozen. They run the successor against it through Converse, changing only the model id, and compare label for label with the incumbent. The successor agrees on 97 per cent of the set and flips a cluster of billing-versus-account edge cases, labelling an ambiguous phrase the other way. That is a regression to fix, not to wave through. They sharpen the label definitions in the prompt, add two examples covering those cases, re-run the eval, and the disagreement clears.

Cutover is a flag. Five per cent of traffic goes to the new model id. They watch the classifier’s confidence scores and the downstream correction rate for a few days, widen to twenty-five per cent, then to everything. The old version stays pinned and reachable throughout, kept warm by a scheduled synthetic call so the 15-day inactivity rule does not withdraw it. At any point a bad signal is a flag flip back to a known-good model. Once the successor has held on full traffic, they drop the old version from config, well ahead of the end-of-life date. Total code change: a config value and two prompt examples.

What’s worth remembering

  1. Bedrock reports a model as ACTIVE or LEGACY in modelLifecycle, with legacyTime, publicExtendedAccessTime and endOfLifeTime beside it; after the end-of-life date the model is removed from every Region and calls fail.
  2. The legacy period is at least six months for models launched before 7 September 2026, and either six months or 45 days after that, named on the model card, so read the card before pinning.
  3. Once a model is Legacy you cannot start new fine-tuning jobs on it or create new Provisioned Throughput for it, and an account that has not called it for 15 days can lose access, so keep the rollback path warm.
  4. Converse removes the mechanical work of switching models, a saved eval set makes the swap a scored decision, and a flagged canary keeps a bad successor to a flag flip.
  5. A fine-tuned model is bound to its base version and usually served by Provisioned Throughput; on-demand custom model deployment covers only a short list of base models in two Regions, while an imported model runs on demand with no base version to deprecate.

These posts are LLM-aided. Backbone, original writing, and structure by Craig. Research and editing by Craig + LLM. Proof-reading by Craig.