Exam-style
A self-hosted model runs on a SageMaker real-time endpoint behind an internal assistant. Requests take anywhere from two seconds to ninety, depending on how many tokens come out. A target-tracking policy on average CPU utilisation has never once triggered, and the 08:15 burst queues for several minutes before capacity arrives. What should the endpoint scale on?
Reveal the answer
D. Target tracking on in-flight concurrency, ConcurrentRequestsPerModel or an equivalent custom concurrency metric, plus a scheduled action that raises the minimum ahead of the known peak
Concurrency is the quantity that tracks what the accelerators are holding, and Little’s Law gives the reason in one line: concurrency equals arrival rate multiplied by mean duration, so on a workload where duration varies by a factor of forty, a fixed request rate maps to wildly different load. Target tracking on ConcurrentRequestsPerModel, or a custom concurrency metric emitted by the container, measures the second factor instead of assuming it. Pair it with a scheduled action that raises the minimum instance count before 08:15, because pulling a multi-gigabyte model onto a fresh accelerator takes minutes and reactive scale-out cannot catch a burst that sharp; the same reasoning is why the floor stays above zero rather than scaling to nothing overnight, which would put a cold start on the first request of the day. Of the others: CPU utilisation measures the idle resource, since generation happens on the GPU and the host CPU marshals bytes, so lowering the target scales on noise rather than load. InvocationsPerInstance is the reflex answer and the reason the first policy failed, since it holds only when duration is roughly constant. A larger instance type raises the ceiling and leaves the shape of the day untouched, so the fleet still sits idle overnight. The Bedrock quota belongs to a different service; nothing here is on-demand Bedrock.
Q. CPU sits at nine per cent, the target-tracking policy never fires, and the morning burst queues for minutes. What should a SageMaker real-time endpoint serving a self-hosted model scale on?
A. In-flight concurrency. Target tracking on ConcurrentRequestsPerModel, or a custom concurrency metric the serving container emits, with a scheduled action that lifts the minimum instance count ahead of the 08:15 peak and a warm floor above zero for the rest of the day. Giving the burst somewhere to wait while instances warm up is the other half of the arrangement.
Why? Concurrency equals arrival rate multiplied by mean duration. A metric built on arrivals alone is right only while duration holds still, and here it runs from two seconds to ninety. CPU utilisation is worse again. The GPU generates and the host CPU marshals bytes, so the reading sits at nine per cent through a burst with every accelerator saturated. Useful utilization monitoring on a GenAI endpoint watches accelerator occupancy and in-flight requests. The auto-scaling configurations that suit this traffic shape pair a concurrency target with a schedule, because model load time makes pure reaction too slow. The capacity planning work then sets the floor and the ceiling: enough warm instances to absorb the first minute of the peak, a maximum that bounds the bill, and a scale-in cooldown long enough that a lull between questions does not evict a model somebody is about to need. Two distractors are worth naming. A bigger instance type raises the ceiling and leaves a day that is silent for eighteen hours exactly as it was. A Bedrock quota increase answers throttling on a managed API, a different failure with a different remedy, and belongs to a serving model this endpoint is not using. Scaling behaviour is part of what picking real-time hosting over serverless or batch commits you to.