Importing Custom Weights into Bedrock

The situation

Clinical research has been fine-tuning Llama 3.1 8B on de-identified medical-notes data for the past quarter. The fine-tune is a LoRALoRAA fine-tuning technique that trains a small low-rank matrix on top of the frozen base model, instead of updating every parameter. adapter merged back into the base weights, trained on 40,000 labelled examples with human-preference signals. The research team’s evaluation shows the fine-tuned model outperforms Claude Sonnet 5 on their specific summarisation task by a noticeable margin on their internal rubric, unsurprising, because the trainingTrainingThe process of fitting a model’s weights to data by minimising a loss function. data is the target distribution.

Training happened on SageMaker training jobs. The weights, roughly 16 GB of safetensors, are in an S3 bucket. Now they have to run in production. The ask: Bedrock’s API surface (the same Converse calls the rest of the stack already uses), the same IAM and VPC posture as the other Bedrock traffic, the same CloudWatch metrics, no SageMaker endpoint for ops to manage, and a predictable bill.

Three questions on the table. First, can Bedrock actually serve these weights, or does the base architecture disqualify them? Second, what’s the throughput and cost model, does it match on-demand foundation models or behave differently? Third, what’s the operational surface for deployment, versioning, and retirement?

What actually matters

Serving custom weights trades configurability for a managed API surface. At one end, a managed-catalog foundation model is ready to go: call the API, pay per tokenTokenThe unit of text an LLM actually sees – usually a short character sequence, not a whole word., done. At the other end, a self-hosted fine-tune is everything configurable, the instance type, the scaling policy, the inferenceInferenceRunning a trained model to produce output – as opposed to training it. code, the container, and everything is the team’s problem. A managed-import path sits in the middle: the platform serves the model, the team brings the weights.

The first thing to ask is what architectures the path supports. Managed-import surfaces only accept weights from a known list of base architectures, in a known format. If the research team has fine-tuned something on that list, the path is open; if they’ve built a novel architecture, it isn’t, and the answer drifts toward self-hosting or a fully managed endpoint.

The second is throughput and cost model. A pay-per-token foundation model and a dedicated-capacity model behave very differently as utilisation changes. Pay-per-token is cheap when traffic is sporadic and expensive when traffic is heavy and constant. Dedicated capacity is cheap per-token at high utilisation and expensive per-token at low utilisation, because the bill ticks regardless of how many calls land on it. Whichever path the workload takes, the shape of the bill follows from that choice.

The third is cold start and scaling. A hosted model that’s been idle has to be brought back online before the next call returns; that’s measurable seconds of latency. Whether that matters depends on whether the workload is interactive or batch, and whether the scaling unit is a request or a slab of capacity.

The fourth is versioning and deployment. Every new weight set is a new model identity somewhere, a new endpoint, a new model ARN, a new container tag. Rolling from v1 to v2 is at minimum a caller-config change; rollback is the same operation in reverse. Whatever the path, the work is making that flip fast and reversible.

The fifth is operational surface compared to alternatives. Self-hosting gives full control and full operational responsibility. A managed import path moves hosting to AWS and removes control over the inference container, the batching strategy and the instance type. A team shipping a fine-tune with no GPU ops to run will take that trade; a team with GPU-ops experience and unusual serving requirements will not.

The sixth is compliance fit. Medical notes: PHI, HIPAA, audit, the works. Whichever path is chosen, the data-handling story has to carry over, no training on inference data, no inference logging outside the account, private network egress, full audit trail.

What we’ll filter on

1. Base-model support, does this path accept the architecture in use?
2. Operational surface, what are we running vs what AWS runs?
3. Cost shape, per-token, per-hour, per-CMU-minute?
4. Latency and cold-start, first-call and steady-state?
5. Version and rollback, how fast from weights-in-S3 to traffic flowing?

The landscape

1. Bedrock Custom Model Import. Upload weights to S3; create an imported model in Bedrock; call it with InvokeModel, InvokeModelWithResponseStream, Converse or ConverseStream using the model ARN. Bedrock hosts the model and scales the number of running copies. The supported architectures are Mistral, Mixtral, Flan, Llama 2 through Llama 3.3 and Mllama, GPTBigCode, the Qwen2, Qwen2.5 and Qwen3 families, and GPT-OSS. It runs in us-east-1, us-east-2, us-west-2 and eu-central-1 only, and cannot be used with Bedrock batch inference or with CloudFormation. Billed per Custom Model Unit per minute. Same IAM, CloudWatch and VPC endpoint story as the rest of the Bedrock stack.

2. SageMaker real-time endpoint. Deploy the model behind a SageMaker endpoint on a chosen instance type (ml.g5, ml.g6, ml.p4d/p5 for larger models). Full control over the inference container, TorchServe or Triton or LMI. Scaling via SageMaker’s autoscaling policies. Billed by instance-hours. Requires endpoint ops, health checks, deployment pipelines, scaling policies, version alias management.

3. SageMaker serverless inference. Pay per invocation with automatic scaling to zero, which sounds right for a trickle of research traffic. It is not available for this model: serverless endpoints exclude GPUs, cap endpoint memory at 6144 MB and cap the container image at 10 GB, so 16 GB of weights on GPU hardware will not run on one.

4. SageMaker JumpStart pre-trained. If the task can be done with a JumpStart model instead of a bespoke fine-tune, it cuts out the training step. Not applicable here, where the training data is what makes the model worth building.

5. Self-hosted on EKS/EC2 with vLLM or TGI. The team’s own GPU cluster running vLLM or Text Generation Inference, exposed via an internal endpoint. Maximum control; maximum operational cost. Correct for teams with GPU-ops maturity and workloads big enough to justify dedicated hardware.

6. Bedrock fine-tuning on a foundation model. Bedrock fine-tunes the Nova family and Nova Canvas in us-east-1, and Claude 3 Haiku plus Llama 3.1, 3.2 and 3.3 in us-west-2, then serves the result. How the result bills depends on the base. Nova Micro, Nova Lite, Nova Pro, Nova 2 Lite and Llama 3.3 70B Instruct can be deployed for on-demand inference and charged per token, provided the model was customised on or after 16 July 2025. Every other base, Llama 3.1 8B included, serves only through a Provisioned Throughput reservation charged per model unit per hour, busy or idle. Fine-tuning one of the on-demand bases skips the import step entirely.

Evaluation

Side by side

Option	Base support	Ops surface	Cost shape	Latency	Version / rollback
Bedrock Custom Model Import	Listed architectures	Minimal	Per-CMU-minute	Warm: normal; cold: restore delay	Import → caller config flip
SageMaker real-time endpoint	Anything	Heavy	Instance-hours	Warm: low; cold: controllable	Endpoint blue/green
SageMaker serverless inference	CPU only, ≤6 GB	Light	Per-invocation	Cold start variable	Endpoint update
JumpStart	Catalog-limited	Light	Varies	Varies	JumpStart update
Self-hosted EKS + vLLM	Anything	Heaviest	Compute-hours	Ours to tune	Our deployment
Bedrock fine-tuning (native)	Bedrock-native only	Minimal	Per-token (Nova, Llama 3.3 70B) / per-unit-hour (rest)	Native	Deployment or PT flip

For the medical-notes team, with a Llama 3.1 8B fine-tune in S3, Bedrock Custom Model Import is the clean answer: the architecture is supported, the operational surface is minimal, and the API aligns with the rest of the Bedrock stack. The catch is the billing model: Custom Model Units charged per minute of activity, which suits steady traffic better than a trickle.

The import and serving flow

The solution

Pre-import checklist. The weights have to be Hugging Face .safetensors on one of the listed architectures, and Llama 3.1 is on that list. config.json, tokenizer_config.json, tokenizer.json and tokenizer.model travel with them; Bedrock reads those to configure serving, and the chat template has to sit inside tokenizer_config.json because no default one is applied. A text model’s weights must be under 200 GB and its maximum context length under 128K, so 16 GB at 8B is comfortably inside both. Fine-tune against transformers 4.51.3, the version Bedrock supports. The S3 prefix needs to be KMS-encrypted with a key the import-job role can decrypt. Region matters twice over: the job runs in one Region and the model is invocable only there, and only four Regions offer the feature at all.

CreateModelImportJob. A single API call starts the import. The required parameters are jobName, importedModelName, roleArn (a service role with S3 read on the weights bucket and KMS decrypt) and modelDataSource (the S3 URI). There is no base-model parameter; the job detects the architecture from the files. importedModelKmsKeyId and vpcConfig are the optional ones worth setting. The job runs async; poll GetModelImportJob until Status reads Complete, which takes several minutes.

What you get back. An imported-model ARN, readable from the console or from ListImportedModels. That ARN is the modelId passed to InvokeModel or Converse. IAM grants bedrock:InvokeModel on that ARN to whichever principals call it. CloudWatch metrics start accumulating on first invocation, including ModelCopy, which is how many copies are running.

Pricing shape. Billing runs per Custom Model Unit per minute, in 5-minute windows starting from the first successful invocation, and any window an invocation lands in counts as active. Bedrock fixes the units per model copy at import time and reports the number as customModelUnitsPerModelCopy; a Llama 3.1 type model at 8B and 128K sequence length needs 2. A unit is USD$0.05718 a minute in us-east-1 and us-west-2, USD$0.07144 in eu-central-1, plus USD$1.95 a month of storage per unit, so those two units cost USD$3.90 a month to keep registered. Bedrock scales the number of running copies with demand, to a default ceiling of 3 per imported model per account, and shuts a copy down after inactive windows. Steady traffic fills every window it is charged for; 100 scattered calls a day pay for 100 windows they barely use.

Cold starts. Bedrock removes copies that have gone inactive. The next call returns ModelNotReadyException and starts restoration, which takes as long as the model size and on-demand fleet availability dictate; a request is served within 5 minutes or that exception comes back. The SDK retries with exponential backoff by default, and total_max_attempts in the botocore config raises the ceiling. A heartbeat invocation every few minutes will hold a copy up, at the cost of making every 5-minute window billable, which is the same bill as running the copy continuously.

Versioning. Every new weight set is a new import and a new imported-model ARN. The application uses a config entry (or SSM Parameter, or Prompt Management if we’ve put prompts in there) that names the current model ARN. Rollout is updating that entry; rollback is pointing it back. Old imports can be left registered, at USD$1.95 per unit per month of storage, or deleted with DeleteImportedModel.

Comparison to Provisioned Throughput. Both paths end at a custom model behind a Bedrock ARN, which makes the difference easy to miss. Fine-tune Llama 3.1 8B on Bedrock instead of importing it, and the result serves only through a Provisioned Throughput reservation, charged per model unit per hour whether calls arrive or not. AWS publishes hourly rates for some bases and quotes the rest through an account team; the published Llama rows sit at USD$21.18 an hour with no commitment and USD$13.08 on a six-month term, which puts one always-on unit into five figures a month. Import the same weights and only active 5-minute windows are charged, so an overnight gap costs the storage line and nothing else. The reservation suits traffic heavy and flat enough to keep a unit saturated; import suits everything else. Decide before training rather than after, because importing also means running the training environment yourself.

Comparison to SageMaker endpoint. The same 8B model on a SageMaker real-time endpoint needs a GPU instance such as an ml.g5.12xlarge, billed by the instance-hour for as long as the endpoint exists, plus health checks, autoscaling policies, container upgrades and a deployment pipeline. Custom Model Import replaces the instance-hour with an active-window charge and hands the endpoint work to AWS. Compare the two on the workload’s real duty cycle rather than on a headline rate.

Worked example

Morning cold start. 09:00, the batch job starts with 30,000 medical notes to summarise. Nothing has been invoked overnight, so the first call comes back ModelNotReadyException while Bedrock restores a copy, and the SDK’s backoff carries the retries. Once a copy is up, requests settle at roughly a second each for 250-token prompts and 60-token responses, and Bedrock scales out to the default ceiling of 3 copies. The batch finishes in about 45 minutes.

Afternoon trickle. Interactive use through a research notebook: ~200 requests over 6 hours, roughly one every two minutes, so almost every 5-minute window is active and one copy stays up throughout.

Overnight idle. 19:00 to 08:00 next morning: no traffic, no active windows, copies removed. The only line on the bill is storage.

Daily totals. About 30,200 invocations. The batch contributes roughly 270 unit-minutes, three copies of two units each for 45 minutes; the afternoon contributes roughly 720, one copy across six hours of mostly active windows. Just under a thousand unit-minutes at the us-east-1 rate is about USD$57 for the day, against USD$3.90 a month of storage. No endpoints to patch and no scaling policies to tune.

Version update in the afternoon. Research team finishes a new fine-tune at 14:00. They kick off CreateModelImportJob; it completes a few minutes later. Their evaluation suite runs against the new ARN for an hour. At 15:45, staging traffic routes to the new ARN via the config flip; the production flip comes the next morning after overnight validation. Rollback path: flip the config back. Average total time from “new weights” to “production traffic”: half a day, most of which is the eval.

What’s worth remembering

1. Custom Model Import accepts a listed set of architectures: Mistral, Mixtral, Flan, Llama 2 through 3.3, Mllama, GPTBigCode, the Qwen families and GPT-OSS. Embedding models and anything off the list go to SageMaker instead.
2. Billing is per Custom Model Unit per minute, in 5-minute windows from the first successful invocation, not per token. Steady traffic fills the windows it is charged for; a trickle pays for windows it barely uses.
3. Ops surface is near-zero next to a SageMaker endpoint. AWS runs the hosting, the scaling between copies and the availability; we manage the weights and the caller config.
4. A copy removed for inactivity returns ModelNotReadyException on the next call and restores in the background, so retry with backoff. A warming heartbeat holds the copy up and makes every window billable.
5. Bedrock’s own fine-tuning removes the import step, but check how the result serves first. Nova bases and Llama 3.3 70B Instruct can be deployed for on-demand per-token inference; every other base, Llama 3.1 8B included, needs an hourly Provisioned Throughput reservation. Claude 3 Haiku is the one Anthropic base Bedrock fine-tunes.
6. The feature runs in us-east-1, us-east-2, us-west-2 and eu-central-1 only, and does not work with Bedrock batch inference or CloudFormation.