Exam-style
A logistics company runs a customer support assistant on the largest, most capable model its provider offers. The answers are good, and quality scores have held steady for three months. The trouble is volume: about 40,000 conversations a day, a per-call cost well over budget, and a first token slow enough that agents type their own replies. Quality has to hold. Which route fits?
Reveal the answer
B. Model distillation: run the flagship model over a few thousand real support prompts, keep its answers, and train a smaller student model on those pairs
Nothing here is a quality complaint: the scores are steady, and the ask is the same behaviour for less money and less time. Only distillation moves both. Amazon Bedrock Model Distillation runs a teacher model over the prompts you supply and fine-tunes a smaller student model on the responses, as one managed job. What comes back answers support conversations as the teacher did, from a smaller model that costs less per token and runs faster, for that use case. Pre-training builds a foundation model from a large unlabelled corpus and belongs to the provider. Continuous pre-training carries that on with unlabelled company text and teaches the domain’s language, leaving the model the same size; Bedrock no longer lists it as a customisation method, and on AWS it runs on SageMaker AI over tens of billions of tokens. Instruction tuning is fine-tuning on labelled pairs, and changes the shape of a reply. Prompt caching helps alongside, since cache reads are billed below the standard input rate, but it leaves the output tokens and the oversized model alone. A distilled model is a custom model, so serving it means a custom model deployment, limited to a few base models and Regions, or Provisioned Throughput billed hourly, never the shared on-demand pool.
Q. A support assistant on a flagship model answers well and holds its quality scores. At 40,000 conversations a day it costs too much per call and replies too slowly. Which training route fits?
A. Model distillation. Run the flagship teacher over the prompts you supply, keep its answers, and train a smaller student model on those pairs. Amazon Bedrock Model Distillation does the generation and training as one managed job.
Why? Sort the routes by what each changes. Pre-training builds a foundation model from a large unlabelled corpus and belongs to the provider. Continuous pre-training carries that on with unlabelled company text at the same model size, and Bedrock no longer lists it among its customisation methods: on AWS it runs on Amazon SageMaker AI. Instruction tuning is fine-tuning on labelled pairs and changes the shape of a reply, which nobody asked for. Distillation changes model size, moving the teacher’s behaviour on this task into something cheaper and faster to run. Prompt caching helps with the instructions that repeat on every call, but leaves the output tokens and the oversized model alone. Plan the serving: a distilled model is a custom model, so it runs on a custom model deployment or Provisioned Throughput, never the shared on-demand pool.