Exam-style
A self-hosted model runs on a SageMaker real-time endpoint behind an internal assistant. Requests take anywhere from two seconds to ninety, depending on how many tokens come out. A target-tracking policy on average CPU utilisation has never once triggered, and the 08:15 burst queues for several minutes before capacity arrives. What should the endpoint scale on?
Reveal the answer
D. Target tracking on in-flight concurrency, the high-resolution ConcurrentRequestsPerModel metric, plus a scheduled action that raises the minimum ahead of the known peak
Concurrency tracks what the accelerators are holding, and Little’s Law gives the reason in one line: concurrency equals arrival rate multiplied by mean duration, so where duration varies by a factor of forty-five, a fixed request rate maps to wildly different load. SageMaker publishes ConcurrentRequestsPerModel for this, and target tracking takes it as the predefined type SageMakerVariantConcurrentRequestsPerModelHighResolution. It emits every ten seconds where InvocationsPerInstance emits once a minute, and on a streamed response it counts a request until the model sends its last token. Pair it with a scheduled action that raises the minimum instance count before 08:15, because provisioning an instance and loading a multi-gigabyte model takes several minutes, and reactive scale-out cannot catch a burst that sharp. Of the others: CPU utilisation reads the idle resource, since generation runs on the GPU while the host CPU marshals bytes, so lowering the target scales on noise. InvocationsPerInstance is the reflex choice and the reason the first policy never fired, since it holds only where duration is roughly constant. A larger instance type raises the ceiling and leaves the shape of the day untouched, so the fleet still sits idle overnight. The Bedrock quota belongs to a different service; nothing here is on-demand Bedrock.
Q. CPU sits at nine per cent, the target-tracking policy never fires, and the morning burst queues for minutes. What should a SageMaker real-time endpoint serving a self-hosted model scale on?
A. In-flight concurrency. Target tracking on ConcurrentRequestsPerModel, with a scheduled action lifting the minimum instance count ahead of the 08:15 peak. Giving the burst somewhere to wait while instances warm up is the other half.
Why? Concurrency equals arrival rate multiplied by mean duration. A metric built on arrivals alone holds only while duration does, and here it runs from two seconds to ninety. ConcurrentRequestsPerModel counts a streamed request until its last token, and publishes every ten seconds rather than once a minute. CPU utilisation is worse again: the GPU generates while the host CPU marshals bytes, so the reading sits at nine per cent with every accelerator saturated. A variant minimum is at least one instance anyway; dropping to zero needs inference components plus a step scaling policy, and invocations error for minutes until capacity returns. A bigger instance type raises the ceiling and leaves a day that is silent for eighteen hours unchanged. A Bedrock quota increase answers throttling on a managed API, a different failure with a different remedy, not on a model this endpoint hosts itself.