Promoting a Fine-Tuned Model into Production

The situation

A claims team has finished its first serious fine-tune. An open-weight 8B base, adapted on forty thousand adjudicated claim notes, evaluated against eight hundred held-out notes it never saw during training. It summarises a note and proposes a settlement band, and on the held-out set it beats the base model comfortably enough that the business wants it in front of adjusters next month.

What exists right now is a prefix in S3: weight files, a tokenizer config, a metrics JSON the training job wrote, and a Slack thread where somebody says the numbers look good. Nothing about that is deployable and nothing about it is auditable.

Three constraints arrived with the go-ahead. Operations want any bad release reverted inside ten minutes without a code deploy and without waiting for somebody to wake up. Compliance want to be able to take any answer given in the last two years and say which artefact produced it and what its evaluation numbers were when it was approved. And the product roadmap has two more variants coming, one for motor claims and one for property, trained the same way off the same base.

What actually matters

Start with what the deployable unit is, because weights in a bucket are not one. A release needs an identity: something immutable, numbered, and referenceable, that a caller can pin to and an auditor can look up. The identity has to be created at promotion time and never overwritten, which rules out the habit of writing every training run to the same latest/ prefix. It also has to carry more than the weights. The artefact alone does not tell you which container image can serve it, which data it was trained on, what it scored, or whether a human ever agreed it was fit to ship. If those facts live in a wiki page next to the artefact rather than attached to it, they drift within a quarter and the compliance answer becomes archaeology.

Then rollback, and specifically its two separate properties: how long reverting takes, and what reverting costs. Those come apart. A revert that changes one pointer is fast and free. A revert that has to re-provision capacity is slow. Worse, if that capacity was bought as a commitment, holding the old and new versions at once is real money spent on the option to change your mind. There is a third property underneath both: a rollback only works if the previous version still exists and is still running. A path where reverting means rebuilding is not a rollback, it is a second deployment during an incident. And ten minutes unattended means an alarm has to be able to trigger the revert; a human deciding at 3am is not a ten-minute control.

The third thing is fleet shape, which is decided by how the customisation was done rather than by how it will be served. One base plus three small deltas is a different capacity problem to four separate models. If each variant needs its own serving capacity, cost scales with the number of variants whether or not anyone is using them. If the delta is small enough to be attached to a shared base at request time, cost scales with traffic instead, and the third variant is nearly free to keep available.

Last, the caller. Where a model lives decides the shape of the request that reaches it, so moving a model between homes is a change to the application, not just to infrastructure. That deserves to be part of the choice rather than a surprise during the migration. Alongside it sits the end of the life: a model that stops serving still has obligations, because the answers it gave are still on file and somebody will eventually ask which version produced them.

What we’ll filter on

1. Identity: is there an immutable, numbered unit carrying the artefact location, the serving container image, the evaluation metrics, and a recorded human decision?
2. Rollback move and rollback time: what changes to revert, how long does it take, and does the previous version stay alive while the new one proves itself?
3. Unattended rollback: can a metric alarm trigger the revert without a person in the loop?
4. Idle cost: are you paying for held capacity, or only for what you serve?
5. Variants over one base: can a single served base carry several customisations, or does each variant need its own hosting?
6. Caller contract: does landing here change the request and response shape the application sends?

The landscape

There are four homes for a customised model on AWS, and they differ far more in their release mechanics than in their inference quality.

Bedrock Custom Model Import. The artefact is uploaded and becomes a Bedrock model id, called through the same runtime API as every foundation model in the catalogue. There is no endpoint to operate, no instance type to choose, and no scaling policy to write. Serving is metered in custom model units active per minute, with a minimum billable window and a monthly storage charge per imported model; imported models are not on the Provisioned Throughput eligibility list, so there is no reservation to buy and nothing to hold between calls. Capacity scales to zero when traffic stops, and the first call after a quiet spell pays a cold start of tens of seconds. Architecture support is the constraint, since the import path accepts a defined set of families rather than anything you can train. What the import path accepts and what it costs to serve is a decision in its own right.

A Bedrock fine-tune served through Provisioned Throughput. Bedrock trains the customisation itself from a JSONL dataset, and the result is a custom model id in your account. Serving it usually means buying Provisioned ThroughputProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. in model units, because most custom bases are not available for on-demand invocation. That reservation is a real commitment with a term attached, so sizing it against actual throughput matters before anything is promoted, and which customisation technique produced the model in the first place decides whether this home is even available.

A SageMaker AI real-time endpoint fed from the SageMaker Model Registry. The artefact is registered as a versioned model package, and an endpoint is created or updated from an approved version. You choose the instance type, own the inference container, and operate the endpoint, which is more surface than the Bedrock paths and considerably more control. It also accepts any architecture, which is why teams training their own bases end up here. The trade against Bedrock hosting for the same weights is set out in the comparison of the two front doors for one model, and the broader shape of that call in choosing how you host a model at all.

Adapters hot-loaded onto a shared base. The family of parameter-efficient adaptation techniques, of which low-rank adaptation [LoRA]LoRAA fine-tuning technique that trains a small low-rank matrix on top of the frozen base model, instead of updating every parameter. is the one most teams meet first, trains a small set of extra weights and leaves the base untouched. The resulting adapter is megabytes against the base model’s gigabytes. Serving stacks can hold one base in GPU memory and attach a requested adapter per request, so one endpoint answers for several variants. Using adapters for model deployment rather than merging them back into the base is what makes the motor and property variants cheap to keep available: three adapters, one base, one endpoint.

What a model package actually holds

Using the SageMaker Model Registry for versioning is the part of this worth spelling out, because it is where the compliance requirement is satisfied and where the release gate lives.

A model package group is the container for one logical model, so the claims summariser gets one group and every training run for it lands inside. Each run registers a model package, numbered by the registry, and that version number is the immutable release identity nothing else in the story provides. The package carries the artefact URI in S3, the URI of the inference container image that can serve it, the evaluation metrics from the run, and a model card describing intended use, training data, and known limitations.

It also carries an approval status, which is the field everything hangs off. A newly registered package sits at PendingManualApproval. A reviewer, or an automated evaluation that passed its thresholds, moves it to Approved. A package that failed review is set to Rejected and stays in the registry as a record of a version that was considered and refused. That status is what automated deployment pipelines to update models read: the deployment step is conditional on an approved package, and it creates or updates an endpoint from the version it finds. That gate is why the registry is not merely a filing cabinet. Nothing reaches an endpoint that a person or a test did not mark approved, and the record of who marked it survives the release.

The vocabulary trap

Both stacks use the word “version” for different things, and mixing them up is a reliable source of confusion. The Model Registry versions model packages, numbered per group. Bedrock versions model ids, and separately versions prompts and agent aliases, which are release units of a different kind entirely and are covered in the treatment of prompt and model versioning. A team that has an approved model package version 7 and a Bedrock custom model id in the same architecture needs both names in its release notes, because neither one identifies the other.

Evaluation

Side by side

Two columns carry the decision. The approval column is empty for both Bedrock rows because there is no package status a pipeline can gate on. A custom model id exists or it does not; any gate has to be built around it in your own tooling. The unattended-rollback column is empty for the same two rows because repointing a model id is a change to your application configuration, and whether an alarm can make that change is a property of your deployment system rather than of Bedrock.

The idle-capacity column reads worse for SageMaker than it deserves. An endpoint runs instances whether or not anyone calls it, which is a genuine cost, but the adapter row turns that from a per-variant cost into a per-fleet one. Three variants on three separate custom models are three bills; three adapters on one base is one.

Promotion and rollback, side by side

The four homes separate most sharply on rollback strategies for failed deployments, and the difference is easiest to see with the two paths drawn against each other.

The solution

For this claims team, the artefact goes into the SageMaker Model Registry and serves from a real-time endpoint, with the motor and property variants riding as adapters on the same base. The Bedrock homes lose here on two specifics rather than on general merit: there is no approval status a release gate can read, and three variants would mean three separately served custom models.

Registration. Every training run registers a model package into the claims-summariser model package group, whether or not anyone intends to ship it. The package carries the artefact URI, the inference container image URI, the metrics from the held-out set, and a model card. Registering unconditionally is what makes the registry a record rather than a shipping queue, and it costs nothing to keep a version nobody promoted.

Approval. New packages land at PendingManualApproval. An evaluation job runs the eight hundred held-out notes against the registered package and writes its scores; a reviewer reads the scores and the model card and moves the package to Approved or Rejected. That decision is the release gate, and the deployment step refuses to touch an endpoint for a package in any other state. Reading the training run before it gets this far decides most rejections earlier and more cheaply.

Deployment guardrails. The endpoint update from an approved package runs blue/green: SageMaker stands up a new fleet on the new package and keeps the old fleet running while traffic moves. Traffic shifting is either canary, a small fixed slice first and then the rest, or linear, equal increments on a timer. Both take a baking period, and both take CloudWatch alarms as the auto-rollback condition. If any armed alarm fires during the shift or the bake, SageMaker moves traffic back to the old fleet and terminates the new one. This satisfies the ten-minute unattended requirement without anybody being on call, because the revert is a shift back to a fleet that never stopped serving.

Adapters. The motor and property models train as adapters against a pinned base version, register as their own model packages, and get approved the same way. The serving container holds the base and loads the requested adapter, so all three variants answer from one endpoint and adding a fourth is a package and a route, not a new fleet. The trap here is coupling: an adapter is small and useless without the exact base version it was trained against, so the base version is part of the release and belongs in the model package’s metadata. Bump the base without retraining the adapters and the outputs degrade quietly, which is worse than an error.

Orchestration. The chain from a finished training job to a shifted endpoint is a pipeline, and everything above is designed to be automated: register, evaluate, gate on approval status, update the endpoint under guardrails. The mechanics of building that pipeline sit outside this scenario; what belongs to the artefact is that the pipeline has a status field to read and a package version to deploy.

Retirement

Most release plans stop at the shift to 100%, so lifecycle management to retire and replace models gets improvised. It does not need much. When v8 replaces v7, set v7’s package status to Rejected so no pipeline can pick it up again, and leave the package in the registry. Deleting the endpoint or endpoint variant stops the compute charge. On a Bedrock path the equivalent is deleting the imported model, or letting the throughput commitment lapse at the end of its term, which is a date to diarise rather than a button to press.

What does not get deleted is the artefact and its evaluation set. Two years from now the compliance question is “which model produced this answer and what did it score”, and the answer is the package version, its metrics, and the held-out set those metrics came from. An S3 lifecycle policy moving old artefacts to a colder storage class is fine; a policy expiring them is how the audit trail is lost.

What the caller sends

Moving a model between these homes changes the request, because the two runtimes take different shapes. The mechanics of the two contracts are worked through where the tool-calling loop meets them; what belongs to a promotion decision is who owns each contract and what pins it to a release.

The SageMaker one is yours, and it travels with the package. The handler that parses the request body lives in the inference container, and that image URI is recorded on the model package, so approving version 7 approves its request shape alongside its weights. Change the handler and you have a new package version and a new endpoint update, which puts a caller-breaking change under the same approval gate and the same automatic revert as a weight change. The cost is that nothing publishes the shape for you: it is documented where your team documents it, or nowhere.

{
  "inputs": ["Claim 88213: vehicle written off, third party admitted fault..."],
  "parameters": { "max_new_tokens": 512, "temperature": 0.2 }
}

Bedrock’s Converse API takes a fixed structure instead, identical across models: messages with roles and typed content blocks, system for the system instruction, and inferenceConfig for the sampling parameters. None of it is versioned with your model, because none of it is yours to version.

{
  "system": [{ "text": "You summarise claim notes and propose a settlement band." }],
  "messages": [
    { "role": "user", "content": [{ "text": "Claim 88213: vehicle written off..." }] }
  ],
  "inferenceConfig": { "maxTokens": 512, "temperature": 0.2 }
}

A team that starts on SageMaker and later imports the same weights into Bedrock rewrites the request layer, the response parsing, and anything that reasoned about token counts from the old response shape. Worth knowing before the move, not during it. The uniformity of the Bedrock contract is also why swapping one Bedrock model for another can be a configuration change while swapping runtimes is not, and it is one of the trade-offs the Generative AI Lens review will surface if nobody raised it earlier.

Worked example

A release that sticks

Training run 41 finishes and registers as model package version 7 in the claims-summariser group, pointing at its artefact prefix, its serving image, and its metrics. The evaluation job runs the held-out notes and writes a settlement-band accuracy two points above version 6. A reviewer reads the model card, confirms the training window excludes the reopened claims cohort, and sets the package to Approved.

The endpoint update starts. A new fleet comes up on v7 while the v6 fleet keeps serving. Canary shifting sends ten per cent of traffic to v7 and holds it there for fifteen minutes with three alarms armed: endpoint 5xx rate, model latency p99, and a custom metric counting responses the downstream validator rejected. Nothing fires. Traffic moves the rest of the way, the v6 fleet drains and terminates, and the release note records package version 7 and the base version it was trained against.

A release that does not

Run 43 registers as version 9 and gets approved on strong offline numbers. The canary opens at ten per cent, and within four minutes the validator-rejection metric triples. The model has started emitting settlement bands as prose rather than the numeric range the downstream service parses. The offline evaluation scored those answers as correct, because it compared meanings rather than formats.

The alarm fires. SageMaker shifts traffic back to the v8 fleet, which has been running throughout, and terminates the v9 fleet. Total exposure is the four minutes at ten per cent. Nobody deployed anything, nobody was paged, and the evaluation set gets a format check added the same afternoon. Version 9 is set to Rejected, and it stays in the registry with its metrics attached, which is how the next reviewer learns that these offline numbers were not sufficient on their own.

What’s worth remembering

1. A model package version in the SageMaker Model Registry is the immutable release identity: artefact URI, serving image, evaluation metrics, model card, and an approval status of PendingManualApproval, Approved, or Rejected that the deployment step gates on.
2. Rollback differs by home, and that difference decides the choice more often than inference quality does: a SageMaker endpoint update is blue/green with canary or linear traffic shifting and CloudWatch alarms that revert it unattended, while a Bedrock path reverts by repointing a model id, which is quick until reserved throughput has to move with it.
3. LoRA and its relatives let one served base carry several variants, so cost scales with traffic rather than with the number of customisations, at the price of pinning every adapter to the exact base version it was trained against.
4. Retiring a model means rejecting the package version and stopping the compute, not deleting the artefact: the evaluation set and the metrics are what answer an audit question two years later.
5. The registry versions model packages while Bedrock versions model ids, prompts, and agent aliases, so a release note needs both names rather than assuming one identifies the other.
6. Moving a model between SageMaker and Bedrock changes the caller, since one takes a serialised body with an explicit content type parsed by your own handler and the other takes messages, system, and inferenceConfig in a fixed shape.