Exam-style
A logistics company runs a customer support assistant on the largest, most capable model its provider offers. The answers are good: agents accept them, quality scores have held steady for three months, and nobody is asking for better. The trouble is volume. The assistant now handles about 40,000 conversations a day, the per-call cost is well over budget, and the first token takes long enough that agents have started typing their own replies while they wait. Quality has to hold. Which route fits?
Reveal the answer
B. Model distillation: run the flagship model over a few thousand real support prompts, keep its answers, and train a smaller student model on those pairs
Nothing here is a quality complaint. The answers are good, the scores are steady, and the ask is to serve the same behaviour for less money and in less time. Only one of the four training routes moves cost and latency. Model distillation runs a large teacher model over a representative set of real prompts, keeps its responses, and uses those pairs to fine-tune a small student model, a use of transfer learning where what transfers is the teacher’s behaviour on one narrow task. Amazon Bedrock Model Distillation runs the generation and the training as a managed job. What comes back is a smaller model that behaves much like the flagship on support conversations, at a lower per-token price and a faster first token, at the cost of some accuracy outside the slice it was taught. Take the other three in turn. Pre-training is the original run that builds a foundation model from a very large unlabelled corpus, costs millions, and belongs to model providers; a customer never picks it. Continuous pre-training carries on that unlabelled training with a company’s own domain text, which teaches vocabulary and phrasing. It leaves the model exactly as large as it was, so every call still costs what it costs today, and this assistant already reads shipping language well enough. Instruction tuning is fine-tuning on labelled prompt-and-completion pairs, and it changes what a good answer looks like: format, tone, the fields a reply carries. Change nothing about the answers and there is nothing for it to fix. Prompt caching is worth naming, because it is real money and it is cheap to try: the standing instructions and the policy text are the same on every call, and caching them stops the company paying full price to reprocess those input tokens each time. It does nothing about the output tokens, and nothing about a model that is simply larger than this workload needs, so it trims the bill rather than solving it. One consequence to plan for: a distilled model is a custom model, and custom models do not run on the shared on-demand pool. Serving one on Amazon Bedrock means Provisioned Throughput, capacity reserved and billed by the hour whether traffic arrives or not. At 40,000 conversations a day that arithmetic usually works; at a tenth of the volume it often does not, so check it against the current token bill before committing.
Q. A support assistant on a flagship model answers well and holds its quality scores. At 40,000 conversations a day it costs too much per call and replies too slowly. Which training route fits?
A. Model distillation. Run the flagship teacher over a few thousand real prompts, keep its answers, and train a smaller student model on those pairs. Amazon Bedrock Model Distillation does the generation and the training as one managed job.
Why? Sort the four routes by what each one changes. Pre-training builds a foundation model from a very large unlabelled corpus and belongs to the provider. Continuous pre-training carries that on with company text and teaches vocabulary, leaving the model the same size and the bill unchanged. Instruction tuning is fine-tuning on labelled prompt-and-completion pairs and changes what a good answer looks like, which nobody is asking for. Distillation is the one that changes model size, using transfer learning to move the teacher’s behaviour on this narrow task into something cheaper and faster to run. Prompt caching is a sensible partial measure alongside it, since the standing instructions repeat on every call, but it leaves the output tokens and the oversized model where they are. Budget for the serving too: a custom model needs Provisioned Throughput, charged by the hour rather than by the token.