Deciding How Far to Customise a Foundation Model

The situation

A company sells practice-management software to about 1,400 veterinary practices. Last year it added an assistant built on Amazon Bedrock. A nurse at the front desk types a question, and the assistant answers from the practice’s own records and the company’s clinical reference library. It handles roughly 60,000 questions a day.

Three complaints have come back from the practices, and they are not the same complaint. The assistant misreads clinical shorthand, so a note recording “PU/PD, BAR, meds q12h” comes back paraphrased as something a vet would not recognise. It quotes withholding periods from a medicine schedule the regulator replaced eight months ago, which is the complaint that made legal sit up. And it answers at length, hedging and apologising, when what the desk wants is two sentences and a number.

Three teams have proposed three fixes. The applied-science pair want to fine-tune a model on five years of support transcripts. The platform team want to build a retrieval layer over the clinical reference library. One engineer thinks the whole thing is a prompt that nobody has rewritten since the prototype. At the back of the room, somebody has asked how much it would cost to train a veterinary model of their own.

What actually matters

These six options are not a quality ladder with the good answer at the top. They fix different failures, and picking by expense rather than by failure shape is how a team spends a quarter on a training run that leaves the original complaint exactly where it was. AWS calls the training end of this model customization, and the methods Amazon Bedrock currently groups under that name are supervised fine-tuning, reinforcement fine-tuning and distillation.

So sort the complaints first. Three shapes cover almost everything a practitioner will meet.

The first is missing facts. A model’s weights are a snapshot of the text it was trained on, frozen at the moment training stopped. The medicine schedule is not in there because it did not exist yet, and it will be replaced again next year. Training cannot fix a fact that keeps changing; it can only move the snapshot forward and leave you with the same problem on a slower clock.

The second is missing behaviour or format. The model can produce two sentences and a number. Nothing in the prompt asks for that shape, and nothing shows it three examples of a good answer. Behaviour is learnable from examples, and examples can arrive in the prompt or in a training set.

The third is missing vocabulary or domain language. Veterinary shorthand is thin on the ground in a general training corpus, so output on “PU/PD” is less reliable than output on ordinary English. Two things fix that: supply a glossary at call time, or push a great deal more veterinary text through the weights.

Cost then arrives in three separate bills, and they behave nothing alike. A one-off training spend is priced by the tokens processed in the run, paid once, and paid again on every base-model upgrade you want to follow. A per-call token spend is what every extra instruction, example and retrieved passage adds to the input tokens on each request, multiplied here by 60,000 calls a day. A serving spend is the one teams forget: a customised model normally needs Provisioned Throughput, capacity reserved for it and billed by the hour whether the desk asks a question or nobody logs in all weekend. Amazon Bedrock will now serve some custom models on demand instead, and the conditions attached are narrow enough to check before planning around them.

Time to first result closes the list. An afternoon’s prompt work can be measured against real questions by Friday. A training run cannot, and until it has been measured nobody knows whether it was needed. Cheapest and fastest first is how a team finds out which failure it actually has before committing to the expensive fix. It is also the ordering that decides how much of the model’s lifecycle the team ends up owning.

What we’ll filter on

1. Failure shape: does the approach fix missing facts, missing behaviour and format, or missing domain vocabulary?
2. Upfront cost: what has to be paid before a single answer gets better?
3. Per-call cost: what does each of the 60,000 daily questions cost once it is live?
4. Standing serving cost: is there a bill that runs while the system is idle?
5. Data needed: nothing, your documents, labelled prompt-and-completion pairs, or a mountain of raw domain text?
6. Time to first result: hours, days, weeks, or months?

The landscape

Six approaches, in ascending order of what it costs to start.

In-context learning

Everything placed in the prompt teaches the model for the duration of one call. Instructions, a glossary of abbreviations, a worked example or three, the retrieved passage, the customer’s own record: all of it arrives through the same window. Handing over no examples is zero-shot, one is single-shot, several is few-shot, and moving from zero to few is often the whole improvement.

Nothing is trained. The weights are untouched, there is no custom model to deploy, and the effect lasts exactly one call, so the same tokens go in again on the next one. That is the trade: no upfront cost at all, and the highest per-call cost of anything here, because those instructions and examples are input tokens on every single request. Prompt templates keep the wording consistent, Prompt management in Amazon Bedrock gives the template a version and a history instead of leaving it as a string in the codebase, and prompt caching cuts the bill for a large block of instructions that is identical call after call.

Retrieval Augmented Generation (RAG)

Retrieval Augmented Generation (RAG) fetches passages relevant to the question from your own documents at call time, and puts them into the prompt alongside the question. The model then answers from the text supplied in the prompt rather than from its weights. Amazon Bedrock Knowledge Bases handles the machinery: it ingests your documents from Amazon S3, splits them into chunks, turns each chunk into an embedding, and stores the result in a vector store. The supported stores are Amazon OpenSearch Serverless, an OpenSearch Service managed cluster, Amazon S3 Vectors, an Amazon Aurora PostgreSQL cluster, a Neptune Analytics graph, and the third-party options Pinecone, Redis Enterprise Cloud and MongoDB Atlas. Aurora is the PostgreSQL route; a standalone Amazon RDS for PostgreSQL instance is not on the list.

The weights are still untouched. What changes is that a fact becomes current by re-ingesting a document, a job measured in minutes, rather than by training anything. Costs sit in four places: a moderate build, an embedding charge on every ingest, a standing bill for the vector store, and more input tokens per call for the retrieved passages. That third one varies by store, hourly for OpenSearch Serverless or Aurora and per request and per gigabyte for S3 Vectors. A RetrieveAndGenerate response comes back with citations naming the source chunks, which matters a great deal when the answer is a withholding period.

Fine-tuning

Fine-tuning continues training a base model on labelled examples, JSONL records pairing a prompt with the completion you wanted for it. The result is a private custom model that answers the way your examples answered. It is the strongest tool for behaviour, format, tone and task shape, and it lets the prompt get shorter, because the instruction no longer needs repeating on every call. Bedrock also offers reinforcement fine-tuning, where you define reward functions instead of supplying labelled pairs.

Three costs, in descending order of how often they are underestimated. Building the dataset comes first, and it is people rather than compute: someone with clinical judgement writing and checking a few thousand pairs. The default cap on training and validation records combined varies by base model, 10,000 on the Claude Haiku and Llama models and 20,000 on the Amazon Nova models, and Service Quotas will raise it. Then the training run itself, priced by the tokens in the corpus multiplied by the number of epochs. Then serving, plus a monthly storage charge for the custom model. Every base-model upgrade means doing the first two again. Once a base model enters its Legacy period, Bedrock accepts no new fine-tuning jobs against it and no new Provisioned Throughput. Weeks, not days.

Continued pre-training

The same operation on unlabelled text. Instead of pairs, you supply raw domain writing, and the model’s output shifts towards one industry: its abbreviations, its drug names, its habitual phrasing. Bedrock’s customization API still accepts CONTINUED_PRE_TRAINING as a job type. The user guide’s current overview of customization methods lists supervised fine-tuning, reinforcement fine-tuning and distillation only, and the quota tables carry continued pre-training limits for the Titan Text base models alone, so confirm your base model supports it before building a plan on it. There is no labelling effort, which sounds cheaper until you see the volume of raw text the method needs, far more than the few thousand documents most companies think they have. The output is again a custom model with a custom model’s serving bill.

Pre-training from scratch

Data selection and the full pre-training run, in-house, starting from randomly initialised weights. A modern foundation model is pre-trained on trillions of tokens across a cluster of accelerators for months, by people who have done it before. It costs millions. For a company that is not a model provider it is almost never the answer, and it is worth naming so that it can be ruled out with a number rather than a shrug, and so it is clear that the model provider has already done this stage for you.

Model distillation

Model distillation takes a large, capable teacher model, runs your prompts through it, and uses its responses as the training data that fine-tunes a smaller student model. Amazon Bedrock Model Distillation automates the generation and the training, and where it applies its own data-synthesis techniques those teacher inference calls land on your bill at the teacher’s on-demand rates. What comes back is a small model that tracks the big one much more closely on your specific task, at lower inference cost and lower latency, giving up some accuracy in return.

Distillation fixes none of the three failures. It reproduces a teacher that is already answering well and makes it cheaper to run. Distil a model that gets withholding periods wrong and you have a smaller, faster model that gets withholding periods wrong.

Evaluation

Side by side

One tick in the stale-facts column does most of the work here. Retrieval is the only approach that fixes a fact by editing a document, and everything else in that column would need a fresh training run each time the regulator publishes. Read the tone-and-format column next and the two cheap approaches split cleanly: retrieval supplies material and leaves the writing style alone, while prompt instructions and examples change the writing style and supply nothing new. The two complaints therefore have two different answers, which is why the argument in the room had no winner.

The cost columns run in opposite directions, and that is the trade to carry into any of these decisions. Upfront cost climbs as you go down; per-call cost falls. Somewhere there is a crossover where a shorter prompt on a trained model beats a long prompt on a shared one, and where that crossover sits depends entirely on call volume. At 60,000 calls a day it is worth calculating. At 600 it is not.

Domain vocabulary has no column because three rows would carry a qualified tick. A glossary in the prompt fixes it cheaply, retrieval fixes it if the glossary is one of the indexed documents, and continued pre-training fixes it properly and expensively. The cheap versions are worth exhausting first.

Which approach fits which failure

The solution

Two approaches, neither of them a training run.

Put the clinical reference library and the regulator’s medicine schedule behind a Bedrock Knowledge Base, and answer from retrieval. Publication of a new schedule becomes a document swap and a re-ingest, and the assistant is current the same morning. Surface the citations that come back with the generated answer, so a nurse reading a withholding period can see which document it came from and check it, which is a stronger position than a confident sentence with nothing behind it. Sizing the retrieval layer and choosing the model that reads the retrieved passages are separate decisions worth taking on their own.

Fix the tone and the shorthand in the prompt. A system instruction stating the house answer shape (two sentences, the dose with units, a citation line, no apology), three worked examples of good answers, and a short glossary of the abbreviations the desk actually uses. Keep it in Prompt management in Amazon Bedrock so a change to the wording has a version and an author rather than appearing in a deploy diff. Turn on prompt caching for the stable block, provided it clears the model’s cache-checkpoint minimum, because that instruction, those examples and that glossary are identical on all 60,000 calls a day and there is no reason to pay full input-token rate for the same text every time.

Turn down the fine-tune, on the grounds of what it does rather than what it costs. Fine-tuning teaches a model how to answer; it does not reliably install what the answer is. A withholding period taught in a training run can still be contradicted by whatever the weights already contained, with nothing on screen to show which one you got. And a fact the regulator revises makes every revision a new training run, a new evaluation and a new deployment. That is a monthly project to keep one number correct, against a re-ingest that takes minutes.

Three things will bite if they are not planned for. First, serving. A customised model normally needs Provisioned Throughput, capacity reserved and billed by the hour, so a quiet weekend costs the same as a busy Monday. Bedrock will serve some custom models on demand instead, through a custom model deployment, but only in US East (N. Virginia) and US West (Oregon), only for Nova Micro, Nova Lite, Nova Pro, Nova 2 Lite and Llama 3.3 70B Instruct, and only where the model was customised on or after 16 July 2025. Check your base model against that list rather than assuming the hourly bill away. Where it does not qualify, get the per-unit hourly rate from your AWS account team, since AWS does not publish one, and set it against the token bill at your real volume before anyone commits. Second, evaluate before you spend rather than after. Assemble a hundred real questions with answers a vet has approved, score the current assistant against them, and re-score after the prompt change and again after retrieval lands. The cheapest approach that clears that bar is the one to ship, and without the bar there is no way to tell whether the expensive approach was needed. Third, keep the ladder in view rather than declaring the matter closed. If the prompt grows until added context dominates the bill, fine-tuning starts to win on cost, and retrieval and a fine-tuned model working together is a well-trodden combination: retrieval carries the facts, training carries the behaviour. Once the answers are good and the volume is high, model distillation is how the same behaviour gets served for less.

Worked example

The arithmetic that settles the fine-tune argument fits on one slide. Take 60,000 calls a day, an added 1,200 input tokens for the instructions, examples and glossary, and another 2,000 for the retrieved passages. Per-token rates differ by model and Region, and AWS publishes them in US dollars, so the rate below is a round illustrative figure rather than a quote.

added context per call    1,200 + 2,000        = 3,200 input tokens
per day                   60,000 x 3,200       = 192,000,000 tokens
per month (30 days)                            = 5.76 billion tokens
at USD$0.30 per million input tokens           = about USD$1,730 a month

Prompt caching can cut the first line, with a condition on it. A cache checkpoint forms only once the static prefix clears the model’s minimum, and that minimum is 512, 1,024 or 4,096 tokens depending on the model, so a 1,200-token block qualifies on some models and not on others. Tokens read from cache are then billed at the model’s cache-read rate, and on some models tokens written to cache cost more than plain input, so measure the saving rather than assuming it.

Now the other side. A fine-tune needs perhaps 5,000 curated prompt-and-completion pairs, inside the default record cap on every fine-tunable base model, and that is a fortnight of a clinician’s time before any compute runs. Then the custom model needs capacity to serve it. AWS does not publish a Provisioned Throughput rate card. The hourly price depends on the base model, the number of model units, and whether you commit for nothing, one month or six, and the documentation sends you to your account team for the per-unit figure. Get that number before the decision rather than after, because it is an hourly charge that runs at three in the morning on a Sunday. The fine-tune then arrives six weeks later, leaving the medicine schedule exactly as wrong as it was.

The shape matters more than the figures. The cheap approaches convert into a per-call charge that tracks usage and disappears if the feature does. The expensive ones convert into a fixed monthly floor plus a project. At ten times this volume the crossover moves and the calculation is worth redoing, which is why call volume belongs in the discussion alongside the other criteria for choosing a model.

What’s worth remembering

1. Sort the failure before choosing an approach: missing facts, missing behaviour or format, and missing domain vocabulary have different fixes, and only one of the three usually calls for training.
2. Retrieval Augmented Generation (RAG) makes a fact current by re-ingesting a document, so anything that changes on a schedule belongs in retrieval rather than in weights.
3. In-context learning costs nothing upfront and produces no custom model, and it adds input tokens to every call, which makes it the first thing to try and the thing to re-cost as volume grows.
4. Fine-tuning teaches behaviour and format from labelled pairs while continued pre-training teaches domain language from raw text, and both produce a custom model that normally needs Provisioned Throughput billed by the hour, with on-demand serving open only to a short list of base models in two Regions.
5. Pre-training a foundation model from scratch costs millions and takes months, so it belongs to model providers rather than on a customisation shortlist.
6. Model distillation trains a small student on a large teacher’s outputs to cut inference cost and latency, so it applies once the answers are already good rather than as a fix for wrong ones.