The situation
A support-automation assistant has been in production for two quarters. It runs on a base foundation model, a careful system prompt, few-shot examples, and a retrieval step that pulls the relevant knowledge-base articles into context. It works, mostly. But two problems have stopped responding to prompt changes.
The first is format. The downstream ticketing system expects replies in a rigid structure: a one-line resolution summary, a severity tag drawn from a fixed vocabulary, and a JSON block of the fields to update. The model gets it right maybe 85% of the time, and the 15% that drift cause silent failures further down the pipeline. More few-shot examples help a little, then plateau, and each one eats context budget.
The second is vocabulary. The company sells industrial-refrigeration equipment, and the domain is thick with part numbers, model families, and terms of art the base model has clearly never seen at volume. It confuses two compressor lines that share a naming prefix, and no amount of prompt scolding fixes it because the confusion is baked into what the model learned.
There is a pile of assets to work with: 40,000 historically resolved tickets with human-written replies, a 6 GB corpus of service manuals, engineering bulletins, and internal wikis, and a monthly budget that finance is watching. The base model, prompting, and retrieval are already in place. The question is which way to change the model, and what serving the result will cost.
What actually matters
The first thing to pin down is what data you actually have, because it decides which routes are even open. Labelled prompt-completion pairs, an input and the exact output you want back, are the fuel for fine-tuning. A large body of raw, unlabelled domain text is the fuel for continued pre-training. These are not interchangeable. You cannot continued-pre-train your way to a rigid output format, and you cannot fine-tune on documents you have not turned into examples. Most teams have far more unlabelled text than labelled pairs, and curating pairs is the expensive, slow part.
The second is what you are actually trying to fix, because the three routes chase different goals. Fine-tuning changes behaviour: it teaches the model to respond in a particular shape, follow a task reliably, adopt a tone. Continued pre-training changes knowledge: it steeps the model in domain language so the vocabulary and relationships stop being foreign. Distillation changes economics: it transfers the behaviour of a large capable model into a smaller, cheaper, faster one, trading a slice of accuracy for a lower serving bill. Matching the route to the goal matters more than any HyperparameterA training setting you choose before the run (epochs, learning rate, batch size), as opposed to a weight the run learns..
The third is the cost of the training run itself. Fine-tuning a foundation model is a bounded, one-off job priced by tokens processed and EpochOne complete pass over the training dataset – more passes means more chance to shift behaviour, and more chance to memorise.. Continued pre-training over gigabytes of text is a much heavier job, more tokens, more compute, a bigger bill, and it is easy to underestimate. Distillation front-loads work too, because the teacher model has to generate a synthetic training set before the student is ever fine-tuned.
The fourth, and the one that surprises people, is the cost of serving the result. A customised model is not necessarily billed the way its base model was. Some routes leave the bill where it started, per request, scaling with use. Others hand back a capacity reservation charged by the clock, busy or idle, which turns a variable cost into a standing one. Where the weights were trained can matter as much as what was done to them, because the same architecture can bill by the clock or by the minute of use depending on how it arrived. That difference changes the maths completely for a low-traffic workload. A model you invoke a thousand times a day may be far cheaper to run than to host.
One fact sits underneath all of this: customisation and retrieval are not rivals. A fine-tuned model that nails the output format still needs fresh facts fed in at query time, so it almost always keeps its retrieval step. Training bakes in behaviour and vocabulary; retrieval supplies today’s inventory levels and this week’s bulletins. The realistic end state is a customised model that still reads from a knowledge base.
What we’ll filter on
- Data on hand: labelled prompt-completion pairs, or a large volume of unlabelled domain text?
- Goal: reliable task and format behaviour, deeper domain knowledge, or cheaper and faster inference?
- Data volume required, and the effort to curate it into the right shape?
- Training-run cost: a bounded fine-tune, a heavy pre-training pass, or teacher-generation plus a fine-tune?
- Serving cost: does the result bill per request, or as a reservation paid for whether or not it is busy?
- Does it still pair with retrieval for fresh facts?
The landscape
Fine-tuning on labelled pairs
You supply a training set of prompt-completion examples, each an input and the exact output you want, and the training job nudges the model’s weights to reproduce that behaviour.
This is the route for task adaptation, format compliance, tone, and consistency. On Amazon Bedrock, fine-tuning is a managed job: point it at a JSONL dataset in S3, choose a supported base model, set a few hyperparameters (epochs, Learning rateHow far each training step moves the model’s weights – too low and nothing shifts, too high and it lurches past what you wanted., Batch sizeHow many training examples the model sees before each weight update – mostly a stability and throughput dial, not a quality one.), and it produces a custom model. The catch is data: you need enough high-quality, correctly-labelled pairs, hundreds to low thousands is a typical starting range, and their quality caps the result. Garbage pairs teach garbage behaviour. How the custom model that comes out is served depends on the base you picked.
Continued pre-training on unlabelled text
You supply a large corpus of raw domain text, no labels, no input-output structure, just documents, and the job continues the model’s original pre-training objective (predicting the next token) over your data. This teaches domain vocabulary, jargon, entities, and the statistical relationships between them. It does not teach the model to follow a task or emit a format; it makes the domain native. Amazon Bedrock supports continued pre-training as a managed job on supported base models, and it is the heavier, more expensive training route because it chews through far more tokens. It is often a first stage: continued pre-train to install the vocabulary, then fine-tune on a smaller labelled set to install the behaviour.
Model distillation
You start from a large, capable, expensive teacher model and use it to produce a training set (its high-quality answers to a set of prompts), then fine-tune a smaller, cheaper, faster student model on that synthetic set. The student learns to imitate the teacher on your workload for a fraction of the inference cost. Amazon Bedrock Model Distillation automates the awkward middle: you provide prompts (and can let it use your production invocation traffic as the prompt source), it runs the teacher to generate the completions, and it fine-tunes the student for you. The deliberate trade is accuracy for cost and latency: the student is not quite the teacher, but it is dramatically cheaper to serve. The distilled student is a custom model, so it inherits the same serving split as any other Bedrock fine-tune.
Parameter-efficient fine-tuning (LoRA) on SageMaker
When you want more control than the managed Bedrock job gives, SageMaker (including JumpStart) fine-tunes open-weight models directly, and the usual mechanism is parameter-efficient fine-tuning, most commonly LoRA (low-rank adaptation). Rather than updating every weight, LoRA trains small adapter matrices and freezes the base, which cuts the memory and compute of the training run enormously and produces a small adapter you can attach at inference. It is the standard way to fine-tune large models affordably, and it opens up models and knobs Bedrock’s managed path does not expose, at the cost of owning more of the pipeline and the hosting yourself.
Preference tuning (RLHF and friends)
When the goal is alignment, making the model prefer helpful, safe, on-brand answers over merely plausible ones, the tool is preference tuning: reinforcement learning from human feedback (RLHF) and lighter-weight relatives such as direct preference optimisation (DPO). Instead of single correct completions, the training signal is comparisons (this answer is better than that one). It is powerful for polishing behaviour and tone once the basics are right, but it needs preference-labelled data and more machinery, so it is rarely the first move for a task-and-format problem.
Evaluation
Side by side
| Route | Data needed | Teaches | Training cost | Serving on Bedrock | Pairs with RAG |
|---|---|---|---|---|---|
| Fine-tuning (labelled) | Prompt-completion pairs | Task, format, tone | Bounded, per-token | On demand or PT, by base model | ✓ |
| Continued pre-training | Large unlabelled corpus | Domain vocab & knowledge | Heavy, many tokens | On demand or PT, by base model | ✓ |
| Model distillation | Prompts (teacher makes rest) | Cheaper copy of a big model | Teacher-gen + fine-tune | On demand or PT, by base model | ✓ |
| LoRA on SageMaker | Prompt-completion pairs | Task, format (open weights) | Low, adapters only | Self-hosted endpoint | ✓ |
| Preference tuning (RLHF) | Preference comparisons | Alignment, tone | High, extra machinery | Provisioned Throughput | ✓ |
Reading it for this situation, unlabelled manuals plus labelled tickets, a format problem and a vocabulary problem, and a watchful budget, no single route is the whole answer. The format failure calls for fine-tuning; the vocabulary confusion for continued pre-training; and finance wants the serving cost before anything is trained.
The solution
Fine-tuning on labelled pairs is the route for the format failure. The 40,000 resolved tickets are already prompt-completion pairs in spirit: the incoming ticket is the input, the human-written structured reply is the output. Curated down to a few thousand clean, correctly-formatted examples, they teach the model the rigid summary-tag-JSON shape far more reliably than any few-shot prompt, and they free up the context budget those examples were eating. The work is in the curation, not the training: dedupe, strip the pairs where the human reply was sloppy, make sure every completion is in the exact target format, because the model learns the format you show it, warts and all. The output is a custom model, and what it costs to serve depends on the base it was trained from, which is the decision below.
Continued pre-training is the route for the vocabulary confusion. The 6 GB of manuals, bulletins, and wikis is exactly the unlabelled domain text this route consumes. Running it teaches the model that two compressor lines sharing a prefix are distinct things, because it has now seen them used in thousands of real sentences. It will not, on its own, fix the output format, that is not what next-token pre-training does, so the natural pattern is two stages: continued pre-train on the corpus to install the vocabulary, then fine-tune on the labelled tickets to install the behaviour. Budget for it honestly; the pre-training pass over gigabytes is the most expensive training job of the three.
Model distillation is the route finance will raise. If the fine-tuned model works but is a large, pricey base to serve, distillation transfers its behaviour into a smaller student that costs a fraction to run. Amazon Bedrock Model Distillation can use the production traffic as the prompt source, run the capable model as teacher to generate ideal completions, and fine-tune the smaller student automatically. The student will give up a little accuracy; whether that is acceptable is a workload question, measured, not guessed. The payoff is inference cost and latency, which matters most when volume is high enough that per-request savings outweigh whatever the student itself costs to serve.
The serving-cost point deserves its own beat because it decides more than the training choice does, and the answer depends on which model was customised and where.
A custom Nova model is the easy case: it serves on demand at base-model rates, per token, with nothing reserved, so the bill still tracks use. Fine-tune most of the Titan and Llama bases on Bedrock and there is no on-demand path at all, the newer Llama 3.3 70B being the exception that does offer one. The result serves through Provisioned Throughput, a capacity reservation billed per model unit per hour: one Llama 3.1 8B unit is USD$24.00 an hour with no commitment, USD$21.18 on a one-month term, USD$13.08 on six months. That is around USD$17,000 a month for a single unit that bills the same whether or not anything calls it, and it is why the ticket assistant’s traffic profile matters more here than its training bill. For a busy workload the reservation is cheaper per request than on-demand; for a few hundred requests a day it can cost more than staying on a base model with a sharper prompt.
There is a fork worth seeing before training starts, because it closes once you commit. Fine-tune Llama outside Bedrock and bring the weights in through Custom Model Import instead, and Bedrock provisions custom model units automatically and bills them per minute of activity, roughly USD$0.06 to USD$0.07 per unit-minute plus USD$1.95 a month of storage, scaling to zero when idle. The trade is that the training environment becomes yours to run, and Bedrock’s managed fine-tuning is what you give up; what comes back is a bill that follows traffic. On a low-traffic custom model that gap dwarfs anything the training choice costs. The full serving comparison, including where on-demand and batch fit, is in choosing an inference option. Do this maths before training anything: a customised model you cannot afford to host is not a solution.
Worked example
Take the assistant as described and walk the routes.
Start with the format problem alone. Prompting has plateaued at 85%. There are 40,000 labelled pairs available. Goal is behaviour, data is labelled, so the route is fine-tuning. Curate ~3,000 clean tickets into JSONL, run a managed Bedrock fine-tuning job for a few epochs, and the format-compliance rate climbs into the high nineties. Cost of the run is bounded and modest. But the result is a custom model, so before celebrating, price the serving path its base family forces on you and check the daily volume justifies it.
Now add the vocabulary problem. Fine-tuning on 3,000 tickets will not un-confuse the two compressor lines, because the pairs do not contain enough of that language to reteach it, and labelling 6 GB of manuals into pairs is absurd. Goal is knowledge, data is unlabelled, so the route is continued pre-training on the manual corpus first. Then fine-tune the pre-trained model on the tickets. Two jobs, two goals: knowledge, then behaviour. The corpus pass is the expensive one; plan the spend.
Finally, watch the bill. Say the customised model works but sits on a large base that is costly to host at the traffic level involved. Run Amazon Bedrock Model Distillation with the fine-tuned model as teacher and the production prompts as the source, and fine-tune a smaller student. Measure the accuracy drop on a held-out set of tickets. If it holds, the student serves the same workload at lower cost and latency, still through its own Provisioned Throughput, and still reading from the knowledge base at query time for this week’s bulletins. Training changed the behaviour and the vocabulary; retrieval keeps supplying the facts that change too fast to train.
What’s worth remembering
- Your data decides the route. Labelled prompt-completion pairs feed fine-tuning; a large unlabelled corpus feeds continued pre-training. They are not interchangeable, and curating pairs is the slow, expensive part.
- Match the route to the goal. Fine-tuning changes behaviour, continued pre-training changes knowledge, distillation changes economics. Naming the goal first saves the wrong training run.
- Continued pre-training installs vocabulary, not format. It teaches the domain’s language from raw text but will not make the model follow a task, so it usually precedes a fine-tune rather than replacing it.
- Fine-tuning is capped by pair quality. The model learns the format and behaviour you show it, including the sloppy examples, so curation matters more than epoch count.
- Serving cost can dwarf training cost, and what it costs depends on the base. A custom Nova serves on demand per token, as does a fine-tuned Llama 3.3 70B; most other Titan and Llama bases serve only through a Provisioned Throughput reservation billed by the hour, idle or not. On low traffic, hosting can cost more than a sharper prompt on a base model, and importing weights trained elsewhere bills per minute instead. Do the maths before you train.
The two-problem support bot lands on a sequence, not a single route: continued pre-train on the manuals for vocabulary, fine-tune on the tickets for format, and reach for distillation only if the serving bill demands it. The routes are not rivals competing for one slot; they are stages that answer different questions, and the deciding questions are always the same two, what data is on hand and what the change is meant to fix, with the serving cost checked before anything runs.