The situation
A support-automation assistant has been in production for two quarters. It runs on a base foundation model, a careful system prompt, few-shot examples, and a retrieval step that pulls the relevant knowledge-base articles into context. It works, mostly. But two problems have stopped responding to prompt changes.
The first is format. The downstream ticketing system expects replies in a rigid structure: a one-line resolution summary, a severity tag drawn from a fixed vocabulary, and a JSON block of the fields to update. The model gets it right maybe 85% of the time, and the 15% that drift cause silent failures further down the pipeline. More few-shot examples help a little, then plateau, and each one takes up context.
The second is vocabulary. The company sells industrial-refrigeration equipment, and the domain is thick with part numbers, model families, and terms of art that barely appear in general text. The assistant returns one compressor line’s specifications under the name of another, and rewording the prompt does not fix it, because the two names sit too close together in what the base model was trained on.
There is a pile of assets to work with: 40,000 historically resolved tickets with human-written replies, a 6 GB corpus of service manuals, engineering bulletins, and internal wikis, and a monthly budget that finance is watching. The base model, prompting, and retrieval are already in place. What remains is choosing which way to change the model, and pricing what serving the result will cost.
What actually matters
The first thing to pin down is what data you actually have, because it decides which routes are even open. Labelled prompt-completion pairs, an input and the exact output you want back, are what fine-tuning trains on. A large body of raw, unlabelled domain text is what continued pre-training trains on. These are not interchangeable. You cannot continued-pre-train your way to a rigid output format, and you cannot fine-tune on documents you have not turned into examples. Most teams have far more unlabelled text than labelled pairs, and curating pairs is the expensive, slow part.
The second is what you are trying to fix, because the three routes aim at different outcomes. Fine-tuning changes behaviour: it trains the model to respond in a particular shape, follow a task reliably, adopt a tone. Continued pre-training changes knowledge: more next-token training over your domain text, so the domain’s terms and the relationships between them end up represented in the weights. Distillation changes economics: it transfers the behaviour of a large capable model into a smaller, cheaper, faster one, and the smaller one ends up a little less accurate. Matching the route to the goal matters more than any HyperparameterA training setting you choose before the run (epochs, learning rate, batch size), as opposed to a weight the run learns..
The third is the training run itself: what it costs, and whose infrastructure it happens on. Supervised fine-tuning on Amazon Bedrock is a bounded, one-off managed job, billed by tokens processed, which is corpus tokens multiplied by EpochOne complete pass over the training dataset – more passes means more chance to shift behaviour, and more chance to memorise.. More next-token training over gigabytes of raw text is a far heavier job, and Bedrock no longer offers it as a managed option, so it lands in SageMaker AI with a cluster, a recipe and checkpoints to look after. Distillation front-loads work too: the teacher model generates a synthetic training set before the student is fine-tuned, and those teacher invocations are billed at the teacher’s on-demand rates.
The fourth, and the one that surprises people, is what serving the result costs. A customised model is not necessarily billed the way its base model was. Some custom models deploy for on-demand inference, and the bill still tracks use, per token. Others are reachable only through Provisioned Throughput, a capacity reservation billed per model unit per hour, busy or idle, which turns a variable cost into a standing one. Which of the two you get depends on the base model you customised and where the training ran. That difference changes the maths completely for a low-traffic workload.
One fact sits underneath all of this: customisation and retrieval work together. A fine-tuned model that nails the output format still needs fresh facts fed in at query time, so it almost always keeps its retrieval step. Training installs behaviour and vocabulary; retrieval supplies today’s inventory levels and this week’s bulletins. The realistic end state is a customised model that still reads from a knowledge base.
What we’ll filter on
- Data on hand: labelled prompt-completion pairs, or a large volume of unlabelled domain text?
- Goal: reliable task and format behaviour, deeper domain knowledge, or cheaper and faster inference?
- Data volume required, and the effort to curate it into the right shape?
- Where the training job runs, and what it costs: a managed Bedrock job billed per token, or a SageMaker pipeline you operate?
- Serving path: can the result deploy for on-demand inference, or only through a reservation billed by the hour?
- Does it still pair with retrieval for fresh facts?
The landscape
Fine-tuning on labelled pairs
You supply a training set of prompt-completion examples, each an input and the exact output you want, and the training job adjusts the model’s weights to reproduce that behaviour.
This is the route for task adaptation, format compliance, tone, and consistency. Bedrock calls it supervised fine-tuning and runs it as a managed job: point it at a JSONL dataset in S3, choose a supported base model, set a few hyperparameters (EpochOne complete pass over the training dataset – more passes means more chance to shift behaviour, and more chance to memorise., Learning rateHow far each training step moves the model’s weights – too low and nothing shifts, too high and it lurches past what you wanted., Batch sizeHow many training examples the model sees before each weight update – mostly a stability and throughput dial, not a quality one.), and it produces a custom model. The supported text bases are Amazon Nova Micro, Lite, Pro and Nova 2 Lite, Anthropic Claude 3 Haiku, and the Meta Llama 3.1, 3.2 and 3.3 Instruct models. The constraint is data: you need enough high-quality, correctly-labelled pairs, and their quality caps the result. Garbage pairs teach garbage behaviour. How the resulting custom model is served depends on the base you picked.
Continued pre-training on unlabelled text
You supply a large corpus of raw domain text, no labels, no input-output structure, just documents, and the job continues the model’s original pre-training objective, predicting the next token, over your data. That installs domain vocabulary, jargon, entities, and the statistical relationships between them. It does not train the model to follow a task or emit a format.
Where this runs has changed, and the change is worth knowing. Bedrock’s managed customisation methods are now supervised fine-tuning, reinforcement fine-tuning, and distillation. Continued pre-training is not among them, and the Amazon Titan Text bases that once carried it have left the catalogue. The route now lives in SageMaker AI: continued pre-training recipes for Amazon Nova on SageMaker HyperPod, or JumpStart’s domain-adaptation fine-tuning for open-weight models, which accepts plain CSV, JSON or TXT files of domain text. The weights come back to Bedrock afterwards, through CreateCustomModel for a SageMaker-trained Nova, or Custom Model Import for open weights. It is still usually a first stage rather than a whole answer: install the vocabulary, then fine-tune on a smaller labelled set to install the behaviour.
Model distillation
You start from a large, capable, expensive teacher model and use it to produce a training set, its answers to a set of prompts, then fine-tune a smaller, cheaper, faster student model on that synthetic set. Amazon Bedrock Model Distillation runs the awkward middle: you supply prompts, or point it at your Bedrock invocation logs so real production traffic becomes the source, and it generates the teacher responses and fine-tunes the student.
The teacher and the student have to be a supported pair from Bedrock’s table, which today means Nova Pro or Nova Premier as teacher for a smaller Nova student, and Llama 3.1 405B, Llama 3.1 70B or Llama 3.3 70B as teacher for a smaller Llama student. Distillation is not currently available for Anthropic models on Bedrock. Teacher invocations during data synthesis are billed at the teacher’s on-demand rates, and synthesis can grow the fine-tuning set to at most 15,000 prompt-response pairs. The student ends up slightly less accurate than the teacher, and considerably cheaper and faster to serve.
Parameter-efficient fine-tuning (LoRA) on SageMaker
When you want more control than the managed Bedrock job gives, SageMaker AI, including JumpStart, fine-tunes open-weight models directly, and the usual mechanism is parameter-efficient fine-tuning, most commonly LoRA (low-rank adaptation). Rather than updating every weight, LoRA trains small adapter matrices and freezes the base, which cuts the memory and compute of the training run enormously and produces a small adapter to attach at inference. The same JumpStart path offers both shapes of training: instruction-based fine-tuning on labelled pairs, and domain-adaptation fine-tuning on raw text. You reach models and knobs Bedrock’s managed path does not expose, and you run more of the pipeline and the hosting yourself.
Preference tuning and reinforcement fine-tuning
When the goal is alignment, training the model to rank helpful, safe, on-brand answers above merely plausible ones, the signal is comparisons rather than single correct completions: reinforcement learning from human feedback (RLHF) and lighter relatives such as direct preference optimisation (DPO). Bedrock now runs a managed form of this as reinforcement fine-tuning, where you supply prompts or invocation logs and define reward functions in Lambda or as a model-as-a-judge grader. It is supported on Nova 2 Lite in us-east-1, and on gpt-oss-20B and Qwen3 32B in us-west-2. DPO and PPO are also available as Nova recipes on SageMaker. This polishes behaviour once the basics are right, and it is rarely the first move for a task-and-format problem.
Evaluation
Side by side
| Route | Data needed | Teaches | Where it runs | Serving | Pairs with RAG |
|---|---|---|---|---|---|
| Supervised fine-tuning | Prompt-completion pairs | Task, format, tone | Bedrock managed job | On demand or PT, by base model | ✓ |
| Continued pre-training | Large unlabelled corpus | Domain vocab & knowledge | SageMaker, then import | By how the weights return | ✓ |
| Model distillation | Prompts, or invocation logs | Cheaper copy of a big model | Bedrock managed job | On demand or PT, by student | ✓ |
| LoRA on SageMaker | Pairs or raw text | Task, format, or vocabulary | SageMaker (open weights) | Self-hosted, or Custom Model Import | ✓ |
| Reinforcement fine-tuning | Prompts plus a reward function | Alignment, judged quality | Bedrock managed job | On demand or PT, by base model | ✓ |
Read it for this situation and no single route is the whole answer. There are unlabelled manuals and labelled tickets, a format problem and a vocabulary problem, and a watchful budget. The format failure calls for fine-tuning, the vocabulary confusion for continued pre-training, and finance wants the serving cost before anything is trained.
The solution
Fine-tuning on labelled pairs is the route for the format failure. The 40,000 resolved tickets are already prompt-completion pairs in spirit: the incoming ticket is the input, the human-written structured reply is the output. Curated down to a few thousand clean, correctly-formatted examples, they train the model on the rigid summary-tag-JSON shape far more reliably than any few-shot prompt, and they release the context those examples were occupying. The work is in the curation, not the training. Dedupe, strip the pairs where the human reply was sloppy, and make sure every completion is in the exact target format, because the model learns the format you show it, warts and all.
Continued pre-training is the route for the vocabulary confusion, and it means leaving Bedrock’s managed jobs to do it. The 6 GB of manuals, bulletins, and wikis is exactly the unlabelled domain text this route consumes. Running it teaches the model that two compressor lines sharing a prefix are distinct things, because it has now processed them in thousands of real sentences. On Nova that is a continued pre-training recipe on SageMaker HyperPod; on an open-weight Llama it is JumpStart domain adaptation, which takes the corpus as TXT. Either way the result is a set of weights you then bring back into Bedrock. It will not fix the output format on its own, so the natural pattern is two stages: adapt to the corpus for vocabulary, then fine-tune on the labelled tickets for behaviour. Budget for it honestly, because the corpus pass is the most expensive training job of the three.
Model distillation is the route finance will raise. If the assistant runs on a large base that costs more to serve than the traffic justifies, distillation transfers its behaviour into a smaller student. Bedrock Model Distillation can read the production invocation logs, so the teacher responses already logged become the training set, and the student learns the format they were produced in. The constraint is the pair table: the teacher has to be one of the supported base models, not an arbitrary custom model you fine-tuned earlier. Nova Pro to Nova Lite, or Llama 3.1 405B to Llama 3.1 8B, are the shapes on offer. The student gives up a little accuracy, and whether that is acceptable is a workload question, measured rather than guessed.
The serving path decides more than the training choice does, and it depends on which base was customised and where.
A custom Nova model is the easy case. Nova Micro, Lite, Pro and Nova 2 Lite deploy for on-demand inference in us-east-1, priced the same as base Nova inference, per token, with nothing reserved. Meta Llama 3.3 70B gets the same treatment in us-west-2. Those five are the whole list; every other fine-tunable base is served through Provisioned Throughput. A Llama 3.1 8B or 70B model unit in us-west-2 is USD$24.00 an hour with no commitment, USD$21.18 on a one-month term, and USD$13.08 on six months. At the no-commitment rate that is about USD$17,300 a month for one unit that bills the same whether or not anything calls it, which is why the assistant’s traffic profile matters more here than its training bill. For a busy workload the reservation costs less per request than on demand; for a few hundred requests a day it can cost more than staying on a base model with a sharper prompt. One condition on the easy case: on-demand deployment requires the model to have been customised on or after 16 July 2025.
There is a fork worth seeing before training starts, because it closes once you commit. Adapt or fine-tune Llama outside Bedrock and bring the weights in through Custom Model Import instead, and Bedrock provisions custom model units automatically and bills them in five-minute windows: USD$0.05718 per unit-minute in us-east-1 and us-west-2, plus USD$1.95 per unit per month of storage. It scales to zero after five minutes with no invocations, and cold-starts in tens of seconds when traffic returns. A Llama 3.1 8B model at 128K context needs two units, so around the clock that is roughly USD$4,900 a month, and far less on a workload with gaps in it. You operate the training environment yourself and give up Bedrock’s managed fine-tuning, and the bill then follows traffic. Custom Model Import also rules out batch inference. The full serving comparison, including where on-demand and batch fit, is in choosing an inference option. Do this maths before training anything: a customised model you cannot afford to host is not a solution.
Worked example
Take the assistant as described and walk the routes.
Start with the format problem alone. Prompting has plateaued at 85%. There are 40,000 labelled pairs available. The goal is behaviour and the data is labelled, so the route is fine-tuning. Curate about 3,000 clean tickets into JSONL, run a managed Bedrock fine-tuning job for a few epochs, and the format-compliance rate climbs. The run itself is bounded and modest. The result is a custom model, so before celebrating, check which serving path the base family allows and whether the daily volume justifies it.
Now add the vocabulary problem. Fine-tuning on 3,000 tickets will not separate the two compressor lines, because the pairs do not contain enough of that language, and labelling 6 GB of manuals into pairs is absurd. The goal is knowledge and the data is unlabelled, so the route is continued pre-training on the manual corpus, which means a SageMaker run and an import back into Bedrock. Then fine-tune the adapted model on the tickets. Two jobs, two goals: knowledge, then behaviour. The corpus pass is the expensive one, so plan the spend.
Finally, watch the bill. Say the assistant sits on Nova Pro and the traffic does not justify what that costs to serve. Run Amazon Bedrock Model Distillation with Nova Pro as the teacher, Nova Lite as the student, and the production invocation logs as the source, since those logged responses already carry the target format. Measure the accuracy drop on a held-out set of tickets. If it holds, the distilled Nova Lite deploys for on-demand inference at Nova Lite token rates, and still reads from the knowledge base at query time for this week’s bulletins. Training changed the behaviour and the vocabulary; retrieval supplies the facts that change too fast to train.
What’s worth remembering
- Your data decides the route. Labelled prompt-completion pairs feed fine-tuning; a large unlabelled corpus feeds continued pre-training. They are not interchangeable, and curating pairs is the slow, expensive part.
- Match the route to the goal. Fine-tuning changes behaviour, continued pre-training changes knowledge, distillation changes economics. Naming the goal first saves the wrong training run.
- Bedrock’s managed customisation methods are supervised fine-tuning, reinforcement fine-tuning, and distillation. Continued pre-training runs in SageMaker AI now, as a Nova HyperPod recipe or JumpStart domain adaptation, and the weights return to Bedrock through an import.
- Fine-tuning is capped by pair quality. The model learns the format and behaviour you show it, including the sloppy examples, so curation matters more than epoch count.
- Distillation only works over supported teacher-student pairs, such as Nova Pro to Nova Lite or Llama 3.1 405B to Llama 3.1 8B. The teacher is a base model, not a custom model you fine-tuned earlier.
- Serving cost can dwarf training cost. Custom Nova models and Llama 3.3 70B deploy for on-demand inference at base token rates; every other fine-tunable base needs Provisioned Throughput at USD$24.00 per model unit per hour with no commitment, idle or not. Importing weights trained elsewhere bills per unit-minute instead, so do the maths before you train.
The two-problem support bot lands on a sequence rather than a single route: adapt to the manuals for vocabulary, fine-tune on the tickets for format, and reach for distillation only if the serving bill demands it. The routes are stages that answer different questions, and the deciding questions stay the same two, what data is on hand and what the change is meant to fix, with the serving cost checked before anything runs.