Which Kind of Training a Foundation Model Needs

The situation

A specialist marine insurer writes hull and cargo policies and settles roughly 22,000 claims a year. Eighteen months ago it put an assistant on Amazon Bedrock in front of its claims desk. A surveyor’s report arrives as a PDF, and the assistant drafts a first-pass assessment note that an adjuster then edits and signs.

The cheap improvements have all been made. Policy wordings and the current clause library sit behind a knowledge base, so the assistant answers from documents rather than from memory, and a glossary of about 400 trade terms rides along in the prompt. Two rounds of prompt work have been done. The assistant is right far more often than it was, and the adjusters still rewrite almost every draft, because it produces four flowing paragraphs when the desk wants eight labelled fields with the applicable exclusion stated first. The team has worked up the customisation ladder and reached the rung where the model itself changes.

Four proposals are now in front of the head of claims. Train a marine model of the company’s own. Fine-tune on the 60,000 historical assessment notes in the claims system. Feed the model forty years of surveyor reports so it learns to talk like a surveyor, an archive the data team has since measured at about 145 million tokens once the repeated headers and standard clauses are stripped out. Or take the answers the assistant already produces and train a smaller model to produce them for less. Somebody has also asked whether the adjusters could just rank drafts and have the model learn from that. Each of those is a different job with a different appetite for data, and nobody has said which data the company can actually supply.

What actually matters

Start with what a foundation model already is when it arrives, because that decides what is left to do. Pre-training is the run that builds general capability from an enormous unlabelled corpus, trillions of tokens of ordinary text, on a cluster of accelerators, over months. It is done by the model provider before the model reaches a catalogue, and it costs millions. Nobody in this building will run one. Naming it matters anyway, because everything else on the list is defined by starting from its output rather than from randomly initialised weights.

That reuse has a name. Transfer learning is the idea that knowledge learned on one task can be carried into another as a starting point, instead of learning the second task from nothing. Fine-tuning is an instance of it, and AWS lists transfer learning alongside instruction tuning, adapting models for specific domains, and continuous pre-training as methods for fine-tuning a foundation model. Read that list as one family rather than four rivals: they all continue training a model that has already been pre-trained, and they differ in what they feed it.

Fine-tuning continues training on a comparatively small labelled set, where each item is a prompt paired with the completion you wanted for it. Because the examples are labelled, the model learns the shape of a good answer: the structure, the ordering, the tone, the length, the task. Instruction tuning is the form this usually takes, fine-tuning on instruction-and-response pairs so that a model which merely continues text becomes one that follows an instruction. Hundreds or a few thousand pairs move a model a long way, which is why the expensive part of fine-tuning is people writing and checking examples rather than compute.

Continuous pre-training feeds raw unlabelled text from one domain into a model that has already been pre-trained. AWS also writes it as continued pre-training, and the two names mean the same job. Nothing is labelled, so nothing teaches the model how to answer; what shifts is its grasp of a vocabulary, the drug names or the vessel types or the clause language of one industry. That costs no labelling effort and wants an enormous amount of text, hundreds of millions of tokens, far more than most companies discover they hold once they go and count.

Distillation is the odd one out because it changes neither knowledge nor behaviour. A large, capable teacher model is run over your prompts, and its outputs become the training data for a smaller student model. What comes back behaves much like the teacher on that narrow task at lower cost and lower latency. It changes the model’s size, so it applies once the answers are already good.

Set those four against each other and the discriminator is the data. Labelled pairs teach behaviour, raw domain text teaches vocabulary, a teacher’s outputs teach cheapness, and human rankings teach preference. Adapting models for specific domains is the goal all of this serves, and it names no single technique. Whether it lands on instruction tuning or on continuous pre-training depends on whether the gap is what the model says or how it says it.

What we’ll filter on

1. What the job changes: general capability, behaviour and format, domain vocabulary, or model size.
2. What kind of data it consumes: raw unlabelled text, labelled prompt-and-completion pairs, a teacher model’s outputs, or human preference rankings.
3. How much of that data it needs, and whether the company actually holds that much.
4. How much human labelling effort has to happen before any compute is bought.
5. Who can realistically run it: a model provider, or this team as a customisation job in its own account.
6. What has to be redone when the base model is upgraded.

The landscape

Four training jobs, in the order the model itself would meet them.

Pre-training

Randomly initialised weights, a curated corpus of trillions of tokens scraped and filtered from broad text, and a training run measured in accelerator-months. The output is a base model that can continue text plausibly on almost any subject and follow no instructions at all. This is where general capability comes from, and it is out of reach for anyone who is not a model provider. Worth pricing once so it can be ruled out with a number. Worth naming too, so that “train our own model” gets unpacked in the room rather than nodded through, since it usually turns out to mean fine-tuning.

Fine-tuning, and instruction tuning as its usual form

Take a pre-trained model, hand it labelled examples of the task you care about, and continue training. Each example is a prompt and the completion that should have come back. On Amazon Bedrock this is a model customisation job over a dataset in Amazon S3; on Amazon SageMaker AI it is a training job over an open-weights model from Amazon SageMaker JumpStart. Either way the output is a private custom model that answers the way the examples answered.

Instruction tuning is fine-tuning where the pairs are instructions and their responses, which is how a base model learns to do as it is told at all. The models in a catalogue have usually had this done to them already. A team fine-tuning today is generally teaching a house-specific version of a task the model can already do adequately: this form, these fields, this order, this length. Data volumes are small by training standards and large by human standards. A few hundred to a few thousand pairs is typical, and every one of them needs someone with the domain judgement to say what the right answer was.

Continuous pre-training

The same continuation of training, without labels. You supply raw domain writing, plain text in S3, and the model’s sense of the language moves towards one industry. It fixes the case where a model’s grasp of “general average” or “inherent vice” is thinner than its grasp of ordinary English. It fixes nothing about output format, because nothing in the data says what a good answer looks like.

The volume it wants is the surprise. Nudging a model’s vocabulary is measured in hundreds of millions of tokens of domain text. A company that says it has “decades of documents” often has tens of millions of tokens once boilerplate and repeated headers come out, which is enough to overfit on and not enough to shift a vocabulary. Count before proposing it.

Distillation

Run a large teacher model over a representative set of your prompts, keep its responses, and fine-tune a small student model on those pairs. Amazon Bedrock Model Distillation automates the generation of the responses and the training. The result is a small model that reproduces the teacher’s behaviour on that task at a fraction of the inference cost and latency, giving up some accuracy elsewhere.

The teacher’s quality sets the ceiling exactly. A teacher that writes four flowing paragraphs when the desk wants eight fields produces a student that writes four flowing paragraphs faster and cheaper. Distillation is a cost move applied after quality is settled.

Evaluation

Side by side

Read the labelling column against the volume column and the trade sits in the open. The two jobs that need no labelling are the two that want vastly more data. The one job whose dataset a small team can build by hand is the one that needs a human judgement attached to every item. No route is cheap in people and cheap in data at the same time.

The “what it changes” column decides the rest. Only one row changes what a good answer looks like, and this desk’s complaint is entirely about what a good answer looks like. The vocabulary row would be the answer if the adjusters were reporting that the assistant misread trade language, and they are not, because the glossary in the prompt already closed that gap. Distillation changes neither and would faithfully reproduce today’s wrong-shaped drafts.

Which job the gap calls for

Ordering the gates this way keeps the conversation honest. Capability comes first because “train our own model” collapses under one question about accelerator-months. Vocabulary comes next because it is settled by counting tokens rather than by opinion. Behaviour comes third, and it is where most teams actually land. Cost comes last, since a cheaper copy of an unsatisfactory answer is not progress.

The solution

Run an instruction tuning job on a curated set of prompt-and-completion pairs, where each prompt is a surveyor’s report plus the standing instruction, and each completion is the eight-field assessment note as an adjuster would have written it. That is a fine-tuning job on Amazon Bedrock over a dataset in Amazon S3, and it targets the one thing the adjusters are complaining about: the shape of the answer. Continuous pre-training is off the list because the vocabulary gap is already closed, and the archive’s 145 million tokens would not close it anyway. Distillation is a conversation for after the drafts are good, when 22,000 claims a year makes the inference bill worth attacking.

Preparing the data

This is where the job succeeds or quietly fails, and AWS names five things to get right: data curation, governance, size, labeling and representativeness.

Data curation means choosing examples rather than collecting volume. The 60,000 historical notes are tempting because they exist, and most of them are unusable: notes written before the eight-field form was adopted, notes an adjuster rewrote three times, notes copied from a template with the fields left empty. Pulling all 60,000 into a training set teaches the model the average of every habit the desk has ever had, including the ones it stopped having. Select instead. Ask two senior adjusters to nominate notes they would be happy to see reproduced, and treat everything else as raw material rather than as training data.

Labelling quality and consistency come next. Two adjusters shown the same report should produce completions that agree on structure, on ordering, and on how much detail a field carries. Inconsistent labels teach the model that both versions are acceptable, and it will average them into something that is neither. Write a one-page rubric, have the labellers work through ten reports together before they work alone, and have a third person spot-check a sample.

Size is the number teams most often get backwards. Hundreds of clean, consistent pairs beat thousands of noisy ones, and adding noisy examples to a clean set makes the result worse rather than diluting the noise. Start at 500 to 1,000 pairs, measure, and add more only where measurement says a case is underserved.

Representativeness asks whether the set covers the cases the model will actually meet, in something like the proportions it will meet them. A training set drawn from whatever was easiest to find will over-represent straightforward container-damage claims and under-represent the general average and total-loss cases that take an adjuster longest. The model then performs best on the work that was already cheap. Build the case mix deliberately and check it against last year’s claims by category.

Governance covers where the data came from and what may be done with it. Provenance for every document, and a consent or contractual basis for using a client’s surveyor report in a training run. Personal data removed before anything leaves the bucket, with Amazon Macie finding what a manual pass misses. A retention rule for the dataset itself, and a record of which dataset version produced which custom model. This is the same care any training dataset deserves, and a training run is harder to unwind than an index, because the data is now in the weights.

Then hold back a validation split before training starts, perhaps 10 to 15 per cent, drawn to the same case mix and never shown to the model. Split by claim rather than by note, so the same claim’s near-duplicate drafts cannot land on both sides. Leakage across that boundary produces a model that scores beautifully and disappoints on the desk, and it is the failure that is hardest to spot after the fact.

Where RLHF fits

The suggestion that adjusters rank drafts describes a real technique. Reinforcement learning from human feedback (RLHF) shows people several candidate outputs for the same prompt and asks them to rank them against each other. Those rankings train a separate reward model that learns what people preferred, and the language model is then tuned to score well against that reward model. Because the reward model stands in for a human, preference reaches millions of training steps without anyone sitting there.

What it teaches is helpfulness, tone and harmlessness, the qualities that are easy to recognise and hard to write down as a correct answer. It is how a raw pre-trained model becomes one that answers usefully rather than only plausibly, and the models in the Bedrock catalogue have already had it applied. It is a poor fit for this desk, where the wanted answer can be written down exactly and a labelled pair says it more directly than a ranking does. Keep ranking as the thing the adjusters do during evaluation, and keep the training on pairs. Where retrieval and a trained model end up running together, each carries a different half of the answer: retrieval supplies the facts, training supplies the form.

Worked example

Eight hundred pairs, built over three weeks by two adjusters at two days a week each. The case mix is set against last year’s claim register rather than against what was easy to export:

The last row is the one that would have been dropped by a team collecting whatever came out of the claims system first, and it is the work that costs the most adjuster time per claim.

Each pair goes into the dataset in the format the customisation job expects, one JSON object per line:

{"prompt": "Draft an assessment note from the surveyor report below. Use the eight standard fields in order, state any applicable exclusion first...\n\nSURVEYOR REPORT\n...", "completion": "APPLICABLE EXCLUSION: none identified\nPERIL: wet damage, condensation\nCAUSE: ...\n"}

The instruction sits in the prompt on every pair, identical each time, because that is the instruction the application will send in production and the model should learn the completion that follows it. A hundred and twenty further pairs, drawn to the same mix from claims not represented above, are held back and never trained on. They are scored before and after, by the adjusters, on whether a draft could be signed with no edits, with light edits, or not at all. The number that decides whether the run was worth doing is that first bucket, and it should be measured on the current assistant first so there is something to compare against.

What’s worth remembering

1. Pre-training builds general capability from an enormous unlabelled corpus and belongs to model providers, so “train our own model” almost always turns out to mean fine-tuning.
2. Transfer learning is the umbrella idea that every later job sits under: continue from a pre-trained model’s weights rather than from random ones.
3. Fine-tuning on labelled prompt-and-completion pairs, usually as instruction tuning, changes behaviour and format, while continuous pre-training on raw unlabelled domain text changes vocabulary and teaches nothing about output shape.
4. Distillation trains a small student on a large teacher’s outputs, so it reduces cost and latency and faithfully reproduces whatever the teacher was already doing wrong.
5. Data curation, governance, size, labeling and representativeness decide the result more than any training setting does, and a few hundred consistent pairs outperform thousands of inconsistent ones.
6. Reinforcement learning from human feedback (RLHF) trains a reward model from human rankings and tunes the model against it, which suits helpfulness and tone rather than a task whose right answer can be written down.