The situation
A company sells practice-management software to about 1,400 veterinary practices. Last year it added an assistant built on Amazon Bedrock. A nurse at the front desk types a question, and the assistant answers from the practice’s own records and the company’s clinical reference library. It handles roughly 60,000 questions a day.
Three complaints have come back from the practices, and they are not the same complaint. The assistant misreads clinical shorthand, so a note recording “PU/PD, BAR, meds q12h” comes back paraphrased as something a vet would not recognise. It quotes withholding periods from a medicine schedule the regulator replaced eight months ago, which is the complaint that made legal sit up. And it answers at length, hedging and apologising, when what the desk wants is two sentences and a number.
Three teams have proposed three fixes. The applied-science pair want to fine-tune a model on five years of support transcripts. The platform team want to build a retrieval layer over the clinical reference library. One engineer thinks the whole thing is a prompt that nobody has rewritten since the prototype. At the back of the room, somebody has asked how much it would cost to train a veterinary model of their own.
What actually matters
These six options are not a quality ladder with the good answer at the top. They fix different failures, and picking by expense rather than by failure shape is how a team spends a quarter on a training run that leaves the original complaint exactly where it was. AWS’s vocabulary for this decision is FM customization, and the thing to weigh across the approaches is their cost tradeoffs.
So sort the complaints first. Three shapes cover almost everything a practitioner will meet.
The first is missing facts. A model’s weights are a snapshot of the text it was trained on, frozen at the moment training stopped. The medicine schedule is not in there because it did not exist yet, and it will be replaced again next year. Training cannot fix a fact that keeps changing; it can only move the snapshot forward and leave you with the same problem on a slower clock.
The second is missing behaviour or format. The model is entirely capable of two sentences and a number. Nobody has told it that is what the desk wants, or shown it three examples of what a good answer looks like. Behaviour is learnable from examples, and examples can arrive in the prompt or in a training set.
The third is missing vocabulary or domain language. Veterinary shorthand is thin on the ground in a general training corpus, so the model’s grasp of “PU/PD” is weaker than its grasp of ordinary English. Two things fix that: hand the model a glossary at call time, or push a great deal more veterinary text through its weights.
Cost then arrives in three separate bills, and they behave nothing alike. A one-off training spend is priced by the tokens processed in the run, paid once, and paid again on every base-model upgrade you want to follow. A per-call token spend is what every extra instruction, example and retrieved passage adds to the input tokens on each request, multiplied here by 60,000 calls a day. A serving spend is the one teams forget: a customised model does not sit on the shared on-demand pool, so it needs capacity reserved for it, billed by the hour whether the desk asks a question or nobody logs in all weekend.
Time to first result closes the list. An afternoon’s prompt work can be measured against real questions by Friday. A training run cannot, and until it has been measured nobody knows whether it was needed. That ordering, cheapest and fastest first, is less about frugality than about learning which failure you actually have before committing to the expensive fix. It is also the ordering that decides how much of the model’s lifecycle the team ends up owning.
What we’ll filter on
- Failure shape: does the approach fix missing facts, missing behaviour and format, or missing domain vocabulary?
- Upfront cost: what has to be paid before a single answer gets better?
- Per-call cost: what does each of the 60,000 daily questions cost once it is live?
- Standing serving cost: is there a bill that runs while the system is idle?
- Data needed: nothing, your documents, labelled prompt-and-completion pairs, or a mountain of raw domain text?
- Time to first result: hours, days, weeks, or months?
The landscape
Six approaches, in ascending order of what it costs to start.
In-context learning
Everything placed in the prompt teaches the model for the duration of one call. Instructions, a glossary of abbreviations, a worked example or three, the retrieved passage, the customer’s own record: all of it arrives through the same window. Handing over no examples is zero-shot, one is single-shot, several is few-shot, and moving from zero to few is often the whole improvement.
Nothing is trained. The weights are untouched, there is no custom model to deploy, and the effect lasts exactly one call, which is why it is paid for again on the next one. That is the trade: no upfront cost at all, and the highest per-call cost of anything here, because those instructions and examples are input tokens on every single request. Prompt templates keep the wording consistent, Amazon Bedrock Prompt Management gives the template a version and a history instead of leaving it as a string in the codebase, and prompt caching reduces the bill for a large block of instructions that is identical call after call.
Retrieval Augmented Generation (RAG)
Retrieval Augmented Generation (RAG) fetches passages relevant to the question from your own documents at call time, and puts them into the prompt alongside the question. The model then answers from text it has been handed rather than from memory. Amazon Bedrock Knowledge Bases does the machinery: it ingests documents from Amazon S3, splits them into chunks, turns each chunk into an embedding, and stores the result in a vector store such as Amazon OpenSearch Service, Amazon Aurora, Amazon Neptune or Amazon RDS for PostgreSQL.
The weights are still untouched. What changes is that a fact becomes current by re-ingesting a document, a job measured in minutes, rather than by training anything. Costs sit in three places: a moderate build, an embedding charge on every ingest, an hourly bill for the vector store, and more input tokens per call for the retrieved passages. In exchange the answer can carry a citation, which matters a great deal when the answer is a withholding period.
Fine-tuning
Fine-tuning continues training a base model on labelled examples, pairs of a prompt and the completion you wanted for it. The result is a private custom model that answers the way your examples answered. It is the strongest tool for behaviour, format, tone and task shape, and it lets the prompt get shorter, because the instruction no longer needs repeating on every call.
Three costs, in descending order of how often they are underestimated. Building the dataset comes first, and it is people rather than compute: someone with clinical judgement writing and checking a few thousand pairs. Then the training run itself, priced by tokens processed. Then serving, because the custom model needs its own reserved capacity. Every base-model upgrade means doing the first two again. Weeks, not days.
Continued pre-training
The same operation on unlabelled text. Instead of pairs, you supply raw domain writing, and the model’s general sense of the language shifts towards one industry: its abbreviations, its drug names, its habitual phrasing. AWS also writes this as continuous pre-training. There is no labelling effort, which sounds cheaper until you see the volume of text it wants; nudging a model’s vocabulary takes hundreds of millions of tokens of domain writing, not the few thousand documents most companies think they have. The output is again a custom model with a custom model’s serving bill.
Pre-training from scratch
Data selection and the full pre-training run, in-house, starting from randomly initialised weights. A modern foundation model is pre-trained on trillions of tokens across a cluster of accelerators for months, by people who have done it before. It costs millions. For a company that is not a model provider it is almost never the answer, and it is worth naming so that it can be ruled out with a number rather than a shrug, and so it is clear that somebody already paid for this stage on your behalf.
Model distillation
Model distillation takes a large, capable teacher model, runs your prompts through it, and uses its responses as the training data that fine-tunes a smaller student model. Amazon Bedrock Model Distillation automates the generation and the training. What comes back is a small model that behaves much more like the big one on your specific task, at lower inference cost and lower latency, giving up some accuracy in return.
Read that carefully, because it fixes none of the three failures. Distillation reproduces a teacher that is already answering well and makes it cheaper to run. Distil a model that gets withholding periods wrong and you have a smaller, faster model that gets withholding periods wrong.
Evaluation
Side by side
| Approach | Upfront cost | Per-call cost | First result | Fixes stale facts | Fixes tone and format | Needs labelled data |
|---|---|---|---|---|---|---|
| In-context learning | None | Highest | Hours | ✗ | ✓ | ✗ |
| Retrieval Augmented Generation (RAG) | Moderate | High | Days | ✓ | ✗ | ✗ |
| Fine-tuning | High | Lower, plus standing capacity | Weeks | ✗ | ✓ | ✓ |
| Continued pre-training | High | Lower, plus standing capacity | Weeks | ✗ | ✗ | ✗ |
| Pre-training from scratch | Millions | Lower, plus standing capacity | Months | ✗ | ✓ | ✗ |
| Model distillation | Moderate to high | Lowest, plus standing capacity | Weeks | ✗ | Copies the teacher | ✗ |
One tick in the stale-facts column decides most of this scenario. Retrieval is the only approach that fixes a fact by editing a document, and everything else in that column would need a fresh training run each time the regulator publishes. Read the tone-and-format column next and the two cheap approaches split cleanly: retrieval supplies material and leaves the writing style alone, while prompt instructions and examples change the writing style and supply nothing new. The two complaints therefore have two different answers, which is why the argument in the room had no winner.
The cost columns run in opposite directions, and that is the trade to carry into any of these decisions. Upfront cost climbs as you go down; per-call cost falls. Somewhere there is a crossover where a shorter prompt on a trained model beats a long prompt on a shared one, and where that crossover sits depends entirely on call volume. At 60,000 calls a day it is worth calculating. At 600 it is not.
Domain vocabulary has no column because three rows would carry a qualified tick. A glossary in the prompt fixes it cheaply, retrieval fixes it if the glossary is one of the indexed documents, and continued pre-training fixes it properly and expensively. The cheap versions are worth exhausting first.
Which approach fits which failure
The solution
Two approaches, neither of them a training run.
Put the clinical reference library and the regulator’s medicine schedule behind a Bedrock Knowledge Base, and answer from retrieval. Publication of a new schedule becomes a document swap and a re-ingest, and the assistant is current the same morning. Turn on citations so a nurse reading a withholding period can see which document it came from and check it, which is a stronger position than a confident sentence with nothing behind it. Sizing the retrieval layer and choosing the model that reads the retrieved passages are separate decisions worth taking on their own.
Fix the tone and the shorthand in the prompt. A system instruction stating the house answer shape (two sentences, the dose with units, a citation line, no apology), three worked examples of good answers, and a short glossary of the abbreviations the desk actually uses. Keep it in Amazon Bedrock Prompt Management so a change to the wording has a version and an author rather than appearing in a deploy diff. Turn on prompt caching for the stable block, because that instruction, those examples and that glossary are identical on all 60,000 calls a day and there is no reason to pay full input-token rate for the same text every time.
Turn down the fine-tune, on the grounds of what it does rather than what it costs. Fine-tuning teaches a model how to answer; it does not reliably install what the answer is. A withholding period taught in a training run can still be contradicted by whatever the weights already contained, with nothing on screen to show which one you got. And a fact the regulator revises makes every revision a new training run, a new evaluation and a new deployment. That is a monthly project to keep one number correct, against a re-ingest that takes minutes.
Three things will bite if they are not planned for. First, a customised model does not run on the shared on-demand pool: it needs Provisioned Throughput, capacity reserved and billed by the hour, so a quiet weekend costs the same as a busy Monday. Cost that hourly rate against the token bill at your real volume before anyone commits, because at moderate traffic the reserved capacity is comfortably the more expensive of the two. Second, evaluate before you spend rather than after. Assemble a hundred real questions with answers a vet has approved, score the current assistant against them, and re-score after the prompt change and again after retrieval lands. The cheapest approach that clears that bar is the one to ship, and without the bar there is no way to tell whether the expensive approach was needed. Third, keep the ladder in view rather than declaring the matter closed. If the prompt grows until added context dominates the bill, fine-tuning starts to pay, and retrieval and a fine-tuned model working together is a well-trodden combination: retrieval carries the facts, training carries the behaviour. Once the answers are good and the volume is high, model distillation is how the same behaviour gets served for less.
Worked example
The arithmetic that settles the fine-tune argument fits on one slide. Take 60,000 calls a day, an added 1,200 input tokens for the instructions, examples and glossary, and another 2,000 for the retrieved passages. Prices vary by model and region, so treat the rate below as illustrative rather than a quote.
added context per call 1,200 + 2,000 = 3,200 input tokens
per day 60,000 x 3,200 = 192,000,000 tokens
per month (30 days) = 5.76 billion tokens
at $0.30 per million input tokens = about $1,730 a month
Prompt caching applies to the 1,200-token block, which is byte-identical on every call, and takes a large bite out of the first line. Call the added-context bill somewhere under $1,500 a month, live by Friday.
Now the other side. A fine-tune needs perhaps 5,000 curated prompt-and-completion pairs, which is a fortnight of a clinician’s time before any compute is bought, plus the training run itself. Then the custom model needs reserved capacity to serve it. At an illustrative $20 an hour, that is roughly $14,600 a month, running at three in the morning on a Sunday, and it arrives six weeks later. It also leaves the medicine schedule exactly as wrong as it was.
The shape matters more than the figures. The cheap approaches convert into a per-call charge that tracks usage and disappears if the feature does. The expensive ones convert into a fixed monthly floor plus a project. At ten times this volume the crossover moves and the calculation is worth redoing, which is why call volume belongs in the discussion alongside the other criteria for choosing a model.
What’s worth remembering
- Sort the failure before choosing an approach: missing facts, missing behaviour or format, and missing domain vocabulary have different fixes, and only one of the three usually calls for training.
- Retrieval Augmented Generation (RAG) makes a fact current by re-ingesting a document, so anything that changes on a schedule belongs in retrieval rather than in weights.
- In-context learning costs nothing upfront and produces no custom model, and is paid for in input tokens on every call, which makes it the first thing to try and the thing to re-cost as volume grows.
- Fine-tuning teaches behaviour and format from labelled pairs while continued pre-training teaches domain language from raw text, and both produce a custom model that needs reserved capacity billed by the hour whether traffic arrives or not.
- Pre-training a foundation model from scratch costs millions and takes months, so it belongs to model providers rather than on a customisation shortlist.
- Model distillation trains a small student on a large teacher’s outputs to cut inference cost and latency, so it applies once the answers are already good rather than as a fix for wrong ones.