Auto-Scaling a Model Endpoint for Bursty GenAI Traffic

The situation

An internal assistant serves about 400 engineers and support staff. It answers questions over runbooks, past tickets, and a product wiki, and it runs an open-weight model the team chose deliberately, served from a SageMaker real-time endpoint on GPU instances. The serving choice was made a while back and is not up for review here; the open weights came with the obligation to run the fleet, and that obligation has now arrived.

The traffic has a shape you could set a clock by. From 08:10 the request rate climbs from a couple of calls a minute to around 300 a minute by 08:40, holds until roughly 09:30, drifts down through the afternoon, and spends the night at almost nothing. Most requests are short lookups that generate two or three hundred tokens. A meaningful minority are “summarise this whole ticket thread” calls that generate several thousand and hold an accelerator for well over a minute.

Auto scaling was switched on once, as target tracking on invocations per instance, and it went badly enough that somebody switched it back off. New instances showed up nine to eleven minutes after the alarm fired, by which time the worst of the burst had already been served slowly or not at all. Through all of it the metric looked comfortable while users watched a spinner. The fleet is now pinned at the peak instance count all day. It bills 144 instance-hours a day and does work worth having in about a fifth of them.

What actually matters

Start with what an instance is actually busy with, because that is where the first policy went wrong. In a conventional web tier a request is served in milliseconds, so arrival rate and occupancy are near enough interchangeable and a request-rate target works fine. Token generation is not like that. One request holds an accelerator for seconds to minutes, and how long it holds it depends on how many tokens come out, which nobody knows when the request is admitted. Two mornings with identical arrival rates and different answer lengths put wildly different loads on the same fleet. The quantity that tracks the load is in-flight concurrency, and Little’s Law connects the two: concurrency equals arrival rate multiplied by mean duration. Targeting arrivals means targeting one of those factors while hoping the other stays put, and on a generative workload it does not.

The second thing is that scale-out is slow in a way that dominates everything else. Count the chain: the metric has to be published, the alarm has to breach for its evaluation periods, an accelerated instance has to be provisioned, the container has to be pulled, tens of gigabytes of weights have to be read and loaded onto the GPU, and the runtime has to warm up. Most of that chain is the weight load, and it is measured in minutes. The burst goes from trough to peak in about half an hour. A purely reactive policy cannot win that race, so the burst has to be met some other way: capacity that already exists because the shape was known in advance, or a place for requests to wait while capacity arrives. Reactive scaling then covers the drift those two miss, rather than carrying the whole load.

Third, the binding resource is accelerator memory, not CPU. The weights occupy a fixed slice of GPU memory. The key-value cache grows on top of that with the number of concurrent sequences and the length of their context, so how many requests one instance can hold at once is a memory question with a hard edge. CPU utilisation on a GPU serving box is close to meaningless as a load signal. There is also a knee: below it, more concurrency converts into more tokens per second; above it, the serving container queues internally, throughput per instance flattens, and time-to-first-token climbs. Past that knee, additional concurrency converts into latency rather than work. Capacity planning for token processing requirements starts by measuring where that knee sits, for this model, this instance type, and this prompt shape.

Fourth, scaling in and scaling out are not symmetric. The overnight idle is where the money is, so the temptation is an aggressive scale-in policy. An endpoint that drops to a single instance at 05:50 and has to climb back at 08:15 takes the full model load delay at the worst moment, and a flapping policy takes it repeatedly. The settings should be asymmetric too: react quickly on the way up, slowly on the way down, and hold a floor high enough that the first real request of the morning reaches a warm instance. SageMaker’s defaults point the same way. Both cooldowns default to 300 seconds, and the high-resolution concurrency metrics accelerate scale-out only; scale-in proceeds at standard-metric speed.

What we’ll filter on

1. Signal quality: does the policy scale on something that tracks accelerator occupancy (in-flight concurrency, GPU memory) rather than arrival rate?
2. Burst absorption: what holds requests during the minutes between demand arriving and capacity arriving?
3. Time to usable capacity: how long from the scaling decision to an instance serving tokens?
4. Idle floor: can it shrink to zero overnight, and what happens to the first request after idle?
5. Interaction shape: does the caller hold a synchronous connection, or can it collect a result later?
6. Operational surface: how much of the scaling machinery does the team own and tune?

The landscape

The mechanism underneath almost every option here is Application Auto Scaling, which registers a scalable target (an endpoint variant’s desired instance count, or an inference component’s desired copy count) and drives it from a policy. The choices worth separating are which metric the policy watches and which kind of policy it is.

Target tracking on invocations per instance. The predefined metric SageMakerVariantInvocationsPerInstance is the default and the one the team already tried. One number, almost no configuration, and a good fit for a fleet where every request costs about the same. It counts arrivals, so it does not separate a morning of short lookups from a morning of thread summarisations. On generative traffic it under-provisions when answers are long and over-provisions when they are short, and the failure is hard to spot because the metric looks healthy either way.

Target tracking on concurrency. SageMaker publishes ConcurrentRequestsPerModel at high resolution, and the matching predefined scaling metric, SageMakerVariantConcurrentRequestsPerModelHighResolution, tracks in-flight requests rather than arrivals. That is the number the accelerators are actually holding, so it moves with generation length. These metrics emit every 10 seconds where InvocationsPerInstance emits once a minute, which takes most of the detection lag out of the front of the scale-out chain. They count requests queued inside the container as well, and on a streaming response they count a request until its last token. On an endpoint using inference components the equivalent is SageMakerInferenceComponentConcurrentRequestsPerCopyHighResolution, on the CloudWatch metric ConcurrentRequestsPerCopy, against copy count. Of the available auto-scaling configurations this is the one to start from on a generative workload.

Step scaling. Instead of steering toward a target, step scaling puts CloudWatch alarm bands around the metric and adds a defined number of instances per band, so a large breach produces a large response. Target tracking deliberately smooths, which is right for drift and wrong for a wall of traffic. Step scaling as a second policy on a high-breach alarm gives a fast lane for the burst without giving up the steady-state behaviour of the target-tracking policy.

Scheduled scaling. A scheduled action sets minimum capacity on a cron, so the fleet is already at size when the traffic arrives rather than reacting to it. This is the direct answer to scale-out lag, and it applies whenever the shape is known and stable, which a weekday-morning internal tool is. The cost is paying for warm instances through the shoulder before the peak, which is cheap compared with paying for them all night.

Inference components with scale to zero. An inference component holds one model plus its declared resource requirements, and copies of it scale independently of the endpoint, down to zero copies. Managed instance scaling adds and removes instances underneath as copies need them. Adding a copy to an instance that is already running is much faster than adding an instance, so a fleet with headroom can absorb a step up in seconds rather than minutes. Zero is a separate configuration: MinInstanceCount set to 0 on the variant’s ManagedInstanceScaling, plus a step-scaling policy fired by a NoCapacityInvocationFailures alarm. Coming back takes several minutes, and invocations during it return an error rather than queueing. This is also the shape to be in if a second or third model shows up behind the same endpoint.

SageMaker Serverless Inference. Provisions on demand, scales to zero when idle, bills for the compute a request uses, and offers provisioned concurrency to hold a warm floor against cold starts. The traffic shape here is close to its ideal case. One constraint closes it anyway. GPUs are on the serverless feature exclusion list, and the memory sizes stop at 6144 MB, so a GPU-served open-weight model does not run on it.

SageMaker Asynchronous Inference. A queued endpoint on instance types you choose, GPUs included. Requests point at an S3 payload, land on an internal queue, get processed, and the result is written back to S3 with an optional SNS notification. Payloads go up to 1 GB and processing up to an hour, which covers the long summarisations comfortably. Scaling runs on ApproximateBacklogSizePerInstance as a customized target-tracking metric, and the variant can register with a minimum capacity of zero. Getting off zero on the first request needs a second, step-scaling policy on a HasBacklogWithoutCapacity alarm; without it the endpoint stays down until the backlog exceeds the target value. The queue absorbs a burst by construction, which is the property the synchronous endpoint lacks. The trade is the interaction shape: the caller no longer holds a connection, so the client has to poll or be notified. The asynchronous pattern has its own shape, and it is a change to the application, not just to the scaling policy.

A queue and a worker fleet you own. SQS in front of workers on ECS or Fargate, scaling on queue depth or backlog per worker. Complete control over admission, priority, per-worker concurrency, and the scaling maths. In exchange the team owns the container, the health checks, the deployment, the accelerator scheduling, and every failure mode SageMaker hosting handles on its own.

Batching strategies, of which there are two. Continuous (rolling) batching inside the serving container packs concurrent sequences into shared forward passes, so tokens per second per accelerator rises steeply until memory runs out. That moves the knee, which changes the capacity plan more than any scaling policy changes the fleet, so a fleet sized without it is sized off the wrong curve. Separately, work with nobody waiting on it, nightly re-summarisation of yesterday’s tickets, backfills after a prompt change, belongs in a batch jobBatch inferenceSubmitting a bulk job of model calls to run asynchronously at a lower per-token price, trading immediacy for cost. rather than on the interactive endpoint, which takes that load off the morning entirely.

Not scaling anything. Moving to a managed foundation model on Bedrock on-demand removes the fleet and the policy, replacing the capacity decision with a quota. Provisioned throughput optimization is the Bedrock-side version of the same argument: reserve model unitsProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. to guarantee a floor for the sustained rate, and let on-demand absorb the shoulders, which is a sizing exercise of its own. What goes away with the fleet is the ability to tune the serving stack, and the reason the open weights were chosen in the first place.

Evaluation

Side by side

Option	Tracks occupancy	Absorbs a burst	Time to capacity	Can idle at zero	Synchronous caller	Team owns the machinery
Target tracking on invocations per instance	✗	✗	Minutes	✗	✓	Low
Target tracking on concurrency (high resolution)	✓	✗	Minutes	✗	✓	Low
Step scaling on a breach alarm	✓	✗	Minutes	✗	✓	Low
Scheduled scaling	n/a	✓ (pre-emptively)	Zero at peak	✓ (off-schedule)	✓	Low
Inference components, scale to zero	✓	✓ (spare copies)	Seconds with headroom, minutes from zero	✓	✓	Medium
Serverless Inference	✓	✓	Cold start	✓	✓	Low
Asynchronous Inference	✓	✓ (queue)	Minutes from zero	✓	✗	Low
SQS with ECS or Fargate workers	✓	✓ (queue)	Minutes	✓	✗	High
Bedrock on-demand	n/a	✓ (quota)	None	✓	✓	None

No row absorbs a burst, serves a synchronous caller and scales on occupancy on its own. The answer here is layered rather than picked. And the Serverless Inference row scores well on every column this table measures, which is why the CPU-only, 6144 MB limit belongs written out beside it. A column-by-column comparison recommends Serverless Inference right up until the model does not fit.

The solution

Take the load off first, because every later number gets easier. The nightly re-summarisation and the post-prompt-change backfills move to a batch job, so the interactive fleet stops competing with work nobody is waiting for. Then turn on continuous batching in the serving container and measure, on this model and this instance type, the sustainable tokens per second per instance and the concurrency at which first-token latency starts climbing without throughput following. That measurement is the capacity plan. Everything after it is arithmetic: peak arrival rate times mean duration gives peak concurrency, peak concurrency divided by the per-instance figure gives the instance count, and the same two numbers give the overnight floor.

The steady-state policy is target tracking on the high-resolution concurrency metric. Set the target below the measured knee rather than at it, so the fleet is still inside the useful part of the curve while new capacity is on its way. Minimum capacity stays above zero. Overnight traffic is small but not absent, and a plain variant scalable target cannot register below one instance in any case. Otherwise the first person in at 06:40 waits out a full model load. Maximum capacity is set from the measured peak with headroom, and the two cooldowns are deliberately asymmetric, short on scale-out and long on scale-in, so the fleet climbs quickly and comes down without flapping.

A scheduled action then raises minimum capacity to close to the peak count at 07:45 on weekdays and lowers it again mid-morning. The burst then meets instances that were already warm, and target tracking handles the difference between the forecast and the day. Scheduled actions are set from the CLI or the Application Auto Scaling API rather than the console. Alongside it, a step-scaling policy on a high-breach concurrency alarm adds several instances at once when concurrency runs well past target, which is the case a smoothed target-tracking policy answers one instance at a time: an incident, a launch, an all-hands that sends everyone to the assistant at once. Auto-scaling configurations optimized for GenAI traffic patterns are almost always this combination rather than any single policy.

The long summarisations move to an asynchronous endpoint with its own scaling on backlog, and that separation does two useful things. It stops a 90-second generation sitting in the same concurrency budget as a 3-second lookup, which is what made the concurrency figure so noisy. It also lets that fleet sit at zero instances between jobs. Concurrent model invocation management then closes the loop at the container level: cap the maximum concurrent requests the serving container will admit, so it queues or sheds cleanly instead of thrashing key-value cache memory past the knee, and have callers retry with exponential backoff and jitter when they are turned away. A rejected request handled well is a much better outcome than an accepted one that times out.

What to watch

Utilization monitoring on a generative endpoint means a specific short list, and CPU is not on it. Watch GPUUtilization and GPUMemoryUtilization per instance, because memory is what runs out, and it runs out before the accelerator is busy. Both are summed across the accelerators on the box, so a four-GPU instance reads 0 to 400 per cent; the Normalized variants report 0 to 100, and only where the endpoint hosts inference components. Watch ConcurrentRequestsPerModel against the scaling target, which shows whether the policy is steering or chasing. Watch InvocationsPerInstance alongside it, not as a scaling signal but as the divisor that tells you whether a change in concurrency came from more requests or from longer answers. Split ModelLatency from OverheadLatency so a slow model and a slow endpoint are distinguishable, and track first-token latency separately from total, because a streaming client feels the first and not the second. ModelSetupTime measures compute-launch time on a serverless endpoint, so it is not available here. Where anything scales from zero, alarm on NoCapacityInvocationFailures instead. SageMaker emits it when a request arrives and no instance is active, and it is what the scale-out-from-zero policy hangs off.

Then add the one metric SageMaker does not publish: output tokens served per instance-hour, pushed as a custom metric. Invocations per instance says how many requests the fleet took. Tokens per instance-hour, held up against the tokens per instance-hour the measurement said the fleet could sustain, says how much of the capacity being paid for is doing work. On this endpoint that ratio started somewhere around a fifth, and it is the number that makes the case for the schedule and the floor without anyone needing to read a scaling policy. It also degrades gracefully into a forecast: if the assistant’s user base doubles, the peak concurrency doubles, and the instance count follows from a number that was measured rather than guessed.

Worked example

A Tuesday, with the layered configuration in place.

At 07:45 the scheduled action lifts minimum capacity from 1 to 5. Five instances are warm and idle by 07:53, which costs roughly twenty minutes of instances nobody is using yet.

At 08:12 the first wave lands. Concurrency climbs past the target within two minutes and target tracking adds a sixth instance, which is serving by 08:22. Nothing is queued in the meantime because the five scheduled instances were sized for most of the peak.

At 08:34 an outage notice goes out and the assistant takes a spike three times its normal peak. Concurrency runs well past target and trips the step-scaling alarm’s upper band, which adds three instances in one action rather than one at a time. The container’s admission cap rejects the excess with a retryable response for the four minutes before those instances are live, and the clients back off and come back rather than timing out on a half-open connection.

At 09:35 demand falls away. The long scale-in cooldown means the fleet stays at nine until 10:05 and then steps down gradually, which costs half an hour of instances and avoids paying for a second climb if the afternoon does something unexpected. At 10:15 the scheduled action drops minimum capacity back to 1, and by 11:00 the endpoint is running two instances against light traffic.

Overnight the endpoint sits at one instance rather than six. The asynchronous summarisation endpoint sits at zero and scales out on backlog when someone queues a job. The day bills around 55 instance-hours rather than 144, and the morning is faster than it was at 144, because the capacity is now in the right place at the right time rather than everywhere all the time.

What’s worth remembering

1. An LLM request holds an accelerator for seconds to minutes and its duration varies with output length, so in-flight concurrency, not invocations per instance, is the signal that tracks fleet load: concurrency equals arrival rate multiplied by mean duration.
2. Model load time means reactive scale-out arrives after the burst, so capacity has to be scheduled ahead of a known shape or the burst has to be queued, with target tracking covering the drift between them.
3. Accelerator memory is the binding resource and the concurrency knee is a measured number: past it, throughput per instance flattens while first-token latency climbs.
4. Continuous batching inside the serving container moves that knee further than any scaling policy moves the fleet, so measure with it switched on before sizing anything.
5. Scale out quickly and scale in slowly, and hold a floor above zero wherever the first request of the morning cannot wait out a cold model load.
6. Serverless Inference excludes GPUs and stops at 6144 MB of memory, so however well its scale-to-zero fits the traffic shape, it is not available to a GPU-served open-weight model.