Exam-style
A logistics company runs a customer support assistant on the largest, most capable model its provider offers. The answers are good, and quality scores have held steady for three months. The trouble is volume: about 40,000 conversations a day, a per-call cost well over budget, and a first token slow enough that agents type their own replies. Quality has to hold. Which route fits?
Reveal the answer
B. Model distillation: run the flagship model over a few thousand real support prompts, keep its answers, and train a smaller student model on those pairs
Nothing here is a quality complaint: the scores are steady, and the ask is the same behaviour for less money and in less time. Only distillation moves both. It runs a large teacher model over a few thousand real prompts, keeps the responses, and fine-tunes a small student model on those pairs; Amazon Bedrock Model Distillation does the generation and the training as one managed job. What comes back answers support conversations as the teacher did, at a lower per-token price and a faster first token, and is weaker outside the slice it was trained on. Pre-training builds a foundation model from a large unlabelled corpus and belongs to model providers. Continuous pre-training carries that on with company text and teaches vocabulary, leaving the model the same size and the bill unchanged. Instruction tuning is fine-tuning on labelled prompt-and-completion pairs, and changes the shape of a reply. Prompt caching helps alongside, since tokens read from cache are billed below the standard input rate, but it leaves the output tokens and the oversized model alone. A distilled model is a custom model, so it does not run on the shared on-demand pool: it needs a custom model deployment, limited to a few base models and Regions, or Provisioned Throughput billed hourly.
Q. A support assistant on a flagship model answers well and holds its quality scores. At 40,000 conversations a day it costs too much per call and replies too slowly. Which training route fits?
A. Model distillation. Run the flagship teacher over a few thousand real prompts, keep its answers, and train a smaller student model on those pairs. Amazon Bedrock Model Distillation does the generation and training as one managed job.
Why? Sort the routes by what each changes. Pre-training builds a foundation model from a very large unlabelled corpus and belongs to the provider. Continuous pre-training carries that on with company text and teaches vocabulary, leaving the model the same size and the bill unchanged. Instruction tuning is fine-tuning on labelled pairs and changes the shape of a reply, which nobody is asking for. Distillation changes model size, moving the teacher’s behaviour on this narrow task into something cheaper and faster to run. Prompt caching helps alongside it, since the standing instructions repeat on every call, but it leaves the output tokens and the oversized model where they are. Plan the serving too: a distilled model is a custom model, so it runs on a custom model deployment or on Provisioned Throughput, never the shared on-demand pool.