Fine-Tuning, Continued Pre-Training, or Distillation

The situation

A support-automation assistant has been in production for two quarters. It runs on a base foundation model, a careful system prompt, few-shot examples, and a retrieval step that pulls the relevant knowledge-base articles into context. It works, mostly. But two problems have stopped responding to prompt changes.

The first is format. The downstream ticketing system expects replies in a rigid structure: a one-line resolution summary, a severity tag drawn from a fixed vocabulary, and a JSON block of the fields to update. The model gets it right maybe 85% of the time, and the 15% that drift cause silent failures further down the pipeline. More few-shot examples help a little, then plateau, and each one eats context budget.

The second is vocabulary. The company sells industrial-refrigeration equipment, and the domain is thick with part numbers, model families, and terms of art the base model has clearly never seen at volume. It confuses two compressor lines that share a naming prefix, and no amount of prompt scolding fixes it because the confusion is baked into what the model learned.

There is a pile of assets to work with: 40,000 historically resolved tickets with human-written replies, a 6 GB corpus of service manuals, engineering bulletins, and internal wikis, and a monthly budget that finance is watching. The base model, prompting, and retrieval are already in place. The question is which way to change the model, and what serving the result will cost.

What actually matters

The first thing to pin down is what data you actually have, because it decides which routes are even open. Labelled prompt-completion pairs, an input and the exact output you want back, are the fuel for fine-tuning. A large body of raw, unlabelled domain text is the fuel for continued pre-training. These are not interchangeable. You cannot continued-pre-train your way to a rigid output format, and you cannot fine-tune on documents you have not turned into examples. Most teams have far more unlabelled text than labelled pairs, and curating pairs is the expensive, slow part.

The second is what you are actually trying to fix, because the three routes chase different goals. Fine-tuning changes behaviour: it teaches the model to respond in a particular shape, follow a task reliably, adopt a tone. Continued pre-training changes knowledge: it steeps the model in domain language so the vocabulary and relationships stop being foreign. Distillation changes economics: it transfers the behaviour of a large capable model into a smaller, cheaper, faster one, trading a slice of accuracy for a lower serving bill. Matching the route to the goal matters more than any hyperparameterHyperparameterA training setting you choose before the run (epochs, learning rate, batch size), as opposed to a weight the run learns. .

The third is the cost of the training run itself. Fine-tuning a foundation model is a bounded, one-off job priced by tokens processed and epochsEpochOne complete pass over the training dataset – more passes means more chance to shift behaviour, and more chance to memorise. . Continued pre-training over gigabytes of text is a much heavier job, more tokens, more compute, a bigger bill, and it is easy to underestimate. Distillation front-loads work too, because the teacher model has to generate a synthetic training set before the student is ever fine-tuned.

The fourth, and the one that surprises people, is the cost of serving the result. On Amazon Bedrock, a customised model is not billed the same way as an on-demand base model. You cannot call it per-token off the shared pool. A custom model has to be served through Provisioned Throughput, which reserves capacity by the hour whether or not it is busy. That turns a variable, per-request cost into a standing monthly commitment, and it changes the maths completely for a low-traffic workload. A model you invoke a thousand times a day may be far cheaper to run than to host.

A softer factor sits underneath all of this: customisation and retrieval are not rivals. A fine-tuned model that nails the output format still needs fresh facts fed in at query time, so it almost always keeps its retrieval step. Training bakes in behaviour and vocabulary; retrieval supplies today’s inventory levels and this week’s bulletins. The realistic end state is a customised model that still reads from a knowledge base.

What we’ll filter on

1. Data on hand: labelled prompt-completion pairs, or a large volume of unlabelled domain text?
2. Goal: reliable task and format behaviour, deeper domain knowledge, or cheaper and faster inference?
3. Data volume required, and the effort to curate it into the right shape?
4. Training-run cost: a bounded fine-tune, a heavy pre-training pass, or teacher-generation plus a fine-tune?
5. Serving cost: does the result need Provisioned Throughput, and does the traffic justify a standing reservation?
6. Does it still pair with retrieval for fresh facts?

The customization landscape

1. Fine-tuning on labelled pairs. You supply a training set of prompt-completion examples, each an input and the exact output you want, and the training job nudges the model’s weights to reproduce that behaviour. This is the route for task adaptation, format compliance, tone, and consistency. On Amazon Bedrock, fine-tuning is a managed job: point it at a JSONL dataset in S3, choose a supported base model, set a few hyperparameters (epochs, learning-rate multiplierLearning rateHow far each training step moves the model’s weights – too low and nothing shifts, too high and it lurches past what you wanted. , batch sizeBatch sizeHow many training examples the model sees before each weight update – mostly a stability and throughput dial, not a quality one. ), and it produces a custom model. The catch is data: you need enough high-quality, correctly-labelled pairs, hundreds to low thousands is a typical starting range, and their quality caps the result. Garbage pairs teach garbage behaviour. The custom model that comes out is served via Provisioned Throughput.

2. Continued pre-training on unlabelled text. You supply a large corpus of raw domain text, no labels, no input-output structure, just documents, and the job continues the model’s original pre-training objective (predicting the next token) over your data. This teaches domain vocabulary, jargon, entities, and the statistical relationships between them. It does not teach the model to follow a task or emit a format; it makes the domain native. Amazon Bedrock supports continued pre-training as a managed job on supported base models, and it is the heavier, more expensive training route because it chews through far more tokens. It is often a first stage: continued pre-train to install the vocabulary, then fine-tune on a smaller labelled set to install the behaviour.

3. Model distillation. You start from a large, capable, expensive teacher model and use it to produce a training set (its high-quality answers to a set of prompts), then fine-tune a smaller, cheaper, faster student model on that synthetic set. The student learns to imitate the teacher on your workload for a fraction of the inference cost. Amazon Bedrock Model Distillation automates the awkward middle: you provide prompts (and can let it use your production invocation traffic as the prompt source), it runs the teacher to generate the completions, and it fine-tunes the student for you. The deliberate trade is accuracy for cost and latency: the student is not quite the teacher, but it is dramatically cheaper to serve. The distilled student is a custom model, so it too needs Provisioned Throughput on Bedrock.

4. Parameter-efficient fine-tuning (LoRA) on SageMaker. When you want more control than the managed Bedrock job gives, SageMaker (including JumpStart) fine-tunes open-weight models directly, and the usual mechanism is parameter-efficient fine-tuning, most commonly LoRA (low-rank adaptation). Rather than updating every weight, LoRA trains small adapter matrices and freezes the base, which cuts the memory and compute of the training run enormously and produces a small adapter you can attach at inference. It is the standard way to fine-tune large models affordably, and it opens up models and knobs Bedrock’s managed path does not expose, at the cost of owning more of the pipeline and the hosting yourself.

5. Preference tuning (RLHF and friends). When the goal is alignment, making the model prefer helpful, safe, on-brand answers over merely plausible ones, the tool is preference tuning: reinforcement learning from human feedback (RLHF) and lighter-weight relatives such as direct preference optimisation (DPO). Instead of single correct completions, the training signal is comparisons (this answer is better than that one). It is powerful for polishing behaviour and tone once the basics are right, but it needs preference-labelled data and more machinery, so it is rarely the first move for a task-and-format problem.

Side by side

Route	Data needed	Teaches	Training cost	Serving on Bedrock	Pairs with RAG
Fine-tuning (labelled)	Prompt-completion pairs	Task, format, tone	Bounded, per-token	Provisioned Throughput	✓
Continued pre-training	Large unlabelled corpus	Domain vocab & knowledge	Heavy, many tokens	Provisioned Throughput	✓
Model distillation	Prompts (teacher makes rest)	Cheaper copy of a big model	Teacher-gen + fine-tune	Provisioned Throughput	✓
LoRA on SageMaker	Prompt-completion pairs	Task, format (open weights)	Low, adapters only	Self-hosted endpoint	✓
Preference tuning (RLHF)	Preference comparisons	Alignment, tone	High, extra machinery	Provisioned Throughput	✓

Reading it for this situation, unlabelled manuals plus labelled tickets, a format problem and a vocabulary problem, and a watchful budget, no single route is the whole answer. The format failure wants fine-tuning; the vocabulary confusion wants continued pre-training; the budget wants to know what serving costs before anything is trained.

The picks in depth

Fine-tuning on labelled pairs is the route for the format failure. The 40,000 resolved tickets are already prompt-completion pairs in spirit: the incoming ticket is the input, the human-written structured reply is the output. Curated down to a few thousand clean, correctly-formatted examples, they teach the model the rigid summary-tag-JSON shape far more reliably than any few-shot prompt, and they free up the context budget those examples were eating. The work is in the curation, not the training: dedupe, strip the pairs where the human reply was sloppy, make sure every completion is in the exact target format, because the model learns the format you show it, warts and all. The output is a custom model, and it will need Provisioned Throughput to serve.

Continued pre-training is the route for the vocabulary confusion. The 6 GB of manuals, bulletins, and wikis is exactly the unlabelled domain text this route consumes. Running it teaches the model that two compressor lines sharing a prefix are distinct things, because it has now seen them used in thousands of real sentences. It will not, on its own, fix the output format, that is not what next-token pre-training does, so the natural pattern is two stages: continued pre-train on the corpus to install the vocabulary, then fine-tune on the labelled tickets to install the behaviour. Budget for it honestly; the pre-training pass over gigabytes is the most expensive training job of the three.

Model distillation is the route the budget will raise. If the fine-tuned model works but is a large, pricey base to serve, distillation transfers its behaviour into a smaller student that costs a fraction to run. Amazon Bedrock Model Distillation can use the production traffic as the prompt source, run the capable model as teacher to generate ideal completions, and fine-tune the smaller student automatically. The student will give up a little accuracy; whether that is acceptable is a workload question, measured, not guessed. The payoff is inference cost and latency, which matters most when volume is high enough that per-request savings outweigh the fixed cost of the Provisioned Throughput reservation the student still needs.

The serving-cost point deserves its own beat because it decides more than the training choice does. Every customised model on Bedrock, fine-tuned, continued-pre-trained, or distilled, is served through Provisioned Throughput, a capacity reservation billed by the hour. For a busy workload that is fine; the reservation is cheaper per request than on-demand. For a workload running a few hundred requests a day, a standing hourly reservation can cost more than staying on a base model with a sharper prompt. Do this maths before training anything: a customised model you cannot afford to host is not a solution.

A worked example: the two-problem support bot

Take the assistant as described and walk the routes.

Start with the format problem alone. Prompting has plateaued at 85%. There are 40,000 labelled pairs available. Goal is behaviour, data is labelled, so the route is fine-tuning. Curate ~3,000 clean tickets into JSONL, run a managed Bedrock fine-tuning job for a few epochs, and the format-compliance rate climbs into the high nineties. Cost of the run is bounded and modest. But the result is a custom model, so before celebrating, price the Provisioned Throughput to serve it and check the daily volume justifies the reservation.

Now add the vocabulary problem. Fine-tuning on 3,000 tickets will not un-confuse the two compressor lines, because the pairs do not contain enough of that language to reteach it, and labelling 6 GB of manuals into pairs is absurd. Goal is knowledge, data is unlabelled, so the route is continued pre-training on the manual corpus first. Then fine-tune the pre-trained model on the tickets. Two jobs, two goals: knowledge, then behaviour. The corpus pass is the expensive one; plan the spend.

Finally, watch the bill. Say the customised model works but sits on a large base that is costly to host at the traffic level involved. Run Amazon Bedrock Model Distillation with the fine-tuned model as teacher and the production prompts as the source, and fine-tune a smaller student. Measure the accuracy drop on a held-out set of tickets. If it holds, the student serves the same workload at lower cost and latency, still through its own Provisioned Throughput, and still reading from the knowledge base at query time for this week’s bulletins. Training changed the behaviour and the vocabulary; retrieval keeps supplying the facts that change too fast to train.

What’s worth remembering

1. Your data decides the route. Labelled prompt-completion pairs feed fine-tuning; a large unlabelled corpus feeds continued pre-training. They are not interchangeable, and curating pairs is the slow, expensive part.
2. Match the route to the goal. Fine-tuning changes behaviour, continued pre-training changes knowledge, distillation changes economics. Naming the goal first saves the wrong training run.
3. Continued pre-training installs vocabulary, not format. It teaches the domain’s language from raw text but will not make the model follow a task, so it usually precedes a fine-tune rather than replacing it.
4. Fine-tuning is capped by pair quality. The model learns the format and behaviour you show it, including the sloppy examples, so curation matters more than epoch count.
5. Distillation trades accuracy for cost and latency. A teacher generates the training data, a smaller student is fine-tuned on it, and Amazon Bedrock Model Distillation automates that middle. Measure the accuracy drop before shipping the student.
6. LoRA is the affordable fine-tuning mechanism on SageMaker. It trains small adapters instead of all the weights, cutting training cost and opening up open-weight models the managed Bedrock path does not expose.
7. Preference tuning (RLHF, DPO) is for alignment, not first fixes. It polishes tone and helpfulness from ranked comparisons once the task and format basics are already right.
8. Serving cost can dwarf training cost. Every customised model on Bedrock is served through Provisioned Throughput, a standing hourly reservation. For low traffic, hosting can cost more than a sharper prompt on a base model; do the maths before you train.
9. Customisation still pairs with retrieval. Training bakes in behaviour and vocabulary; a knowledge base supplies the facts that change too fast to retrain, so the realistic end state keeps both.

The two-problem support bot lands on a sequence, not a single route: continued pre-train on the manuals for vocabulary, fine-tune on the tickets for format, and reach for distillation only if the serving bill demands it. The routes are not rivals competing for one slot; they are stages that answer different questions, and the deciding questions are always the same two, what data is on hand and what the change is meant to fix, with the serving cost checked before anything runs.