Exam-style
A platform serves 140 fine-tuned models, one per customer. They differ in size, a handful take steady traffic while most see a few requests a day, and each needs its own scaling behaviour on shared GPUs. What hosting approach fits?
Reveal the answer
C. Inference components, one per model, deployed to a shared endpoint
Inference components decouple hosting from the endpoint: each holds one model with its own compute requirements (CPU cores, accelerators, minimum memory) and its own copy count and scaling, down to zero copies so another component can scale up in its place. That per-model allocation is what handles models of differing sizes with differing traffic on shared accelerators, and each can be updated independently. A multi-model endpoint would be right for a long tail of similar models sharing one framework and container, but it shares memory rather than allocating it, works best when models are similar in size and latency, and gives no per-model scaling. An endpoint per model bills 140 mostly-idle instances, Serverless cannot use GPUs at all, and Batch Transform does not serve on-demand requests.
Q. How do you serve dozens or hundreds of models without an endpoint each?
A. Pack them onto shared infrastructure. Inference components give each model its own CPU, memory, accelerators, copy count, and scaling (down to zero copies), and let you update models one at a time. Multi-model endpoints instead share one serving container across many similar models, loading each into memory on first invocation and unloading the least-used when memory runs short.
Why? Model count is a separate axis from traffic shape. Inference components suit differently-sized models needing independent scaling; multi-model endpoints suit a long tail of same-framework models that can absorb a cold-start penalty.