Exam-style
A platform serves 140 fine-tuned models on SageMaker AI, one per customer. They differ in size, a handful take steady traffic while most see a few requests a day, and each needs its own scaling behaviour on shared GPUs. What hosting approach fits?
Reveal the answer
C. Inference components, one per model, deployed to a shared endpoint
Inference components decouple hosting from the endpoint. Each one holds a single model with its own compute requirements (CPU cores, accelerators, a memory floor), its own copy count, and its own scaling. A component can scale to zero copies, which makes room for another one to scale up. That per-model allocation handles models of differing sizes and differing traffic on shared accelerators, and each model can be updated independently. A multi-model endpoint would be right for a long tail of models that share one framework and serving container, but it time-shares memory rather than allocating it, works best when models are similar in size and invocation latency, and gives no per-model scaling. An endpoint per model means paying for 140 mostly-idle instances, Serverless Inference excludes GPUs, and Batch Transform runs over datasets in S3 rather than serving on-demand requests.
Q. How do you serve dozens or hundreds of models without an endpoint each?
A. Pack them onto shared infrastructure. Inference components give each model its own CPU cores, memory, accelerators, copy count, and scaling, down to zero copies, and let you update models one at a time. Multi-model endpoints instead share one serving container across many models on the same framework. Each model loads into memory the first time an instance is asked for it, and unused models are unloaded when that instance runs short of memory.
Why? Model count is a separate axis from traffic shape. Inference components suit differently-sized models needing independent scaling; multi-model endpoints suit a long tail of same-framework models where occasional cold-start latency is acceptable.