Auto-Scaling a Model Endpoint for Bursty GenAI Traffic

The situation

An internal assistant serves about 400 engineers and support staff. It answers questions over runbooks, past tickets, and a product wiki, and it runs an open-weight model the team chose deliberately, served from a SageMaker real-time endpoint on GPU instances. The serving choice was made a while back and is not up for review here; the open weights came with the obligation to run the fleet, and that obligation has now arrived.

The traffic has a shape you could set a clock by. From 08:10 the request rate climbs from a couple of calls a minute to around 300 a minute by 08:40, holds until roughly 09:30, drifts down through the afternoon, and spends the night at almost nothing. Most requests are short lookups that generate two or three hundred tokens. A meaningful minority are “summarise this whole ticket thread” calls that generate several thousand and hold an accelerator for well over a minute.

Auto scaling was switched on once, as target tracking on invocations per instance, and it went badly enough that somebody switched it back off. New instances showed up nine to eleven minutes after the alarm fired, by which time the worst of the burst had already been served slowly or not at all. Through all of it the metric looked comfortable while users watched a spinner. The fleet is now pinned at the peak instance count all day. It bills 144 instance-hours a day and does work worth having in about a fifth of them.

What actually matters

Start with what an instance is actually busy with, because that is where the first policy went wrong. In a conventional web tier a request is served in milliseconds, so arrival rate and occupancy are near enough interchangeable and a request-rate target works fine. Token generation is not like that. One request holds an accelerator for seconds to minutes, and how long it holds it depends on how many tokens come out, which nobody knows when the request is admitted. Two mornings with identical arrival rates and different answer lengths put wildly different loads on the same fleet. The quantity that tracks the load is in-flight concurrency, and Little’s Law connects the two: concurrency equals arrival rate multiplied by mean duration. Targeting arrivals means targeting one of those factors while hoping the other stays put, and on a generative workload it does not.

The second thing is that scale-out is slow in a way that dominates everything else. Count the chain: the metric has to be published, the alarm has to breach for its evaluation periods, an accelerated instance has to be provisioned, the container has to be pulled, tens of gigabytes of weights have to be read and loaded onto the GPU, and the runtime has to warm up. Most of that chain is the weight load, and it is measured in minutes. The burst goes from trough to peak in about half an hour. A purely reactive policy cannot win that race, so the burst has to be met some other way: capacity that already exists because the shape was known in advance, or a place for requests to wait while capacity arrives. Reactive scaling then covers the drift those two miss, rather than carrying the whole load.

Third, the binding resource is accelerator memory, not CPU. The weights occupy a fixed slice of GPU memory. The key-value cache grows on top of that with the number of concurrent sequences and the length of their context, so how many requests one instance can hold at once is a memory question with a hard edge. CPU utilisation on a GPU serving box is close to meaningless as a load signal. There is also a knee: below it, more concurrency converts into more tokens per second; above it, the serving container queues internally, throughput per instance flattens, and time-to-first-token climbs. Past that knee, additional concurrency converts into latency rather than work. Capacity planning for token processing requirements starts by measuring where that knee sits for this model, this instance type, and this prompt shape, and it produces one number: sustainable tokens per second per instance at an acceptable first-token latency. Everything downstream, the scaling target, the instance count, the forecast, is arithmetic on that number.

Fourth, scale-in has a cost that scale-out does not. The overnight idle is where the money is, so the temptation is an aggressive scale-in policy. But an endpoint that drops to a single instance at 05:50 and has to climb back at 08:15 pays the model load penalty at exactly the moment it can least afford it, and a flapping policy pays it repeatedly. Scaling out and scaling in are asymmetric decisions and should carry asymmetric settings: react quickly on the way up, slowly on the way down, and keep a floor high enough that the first real request of the morning meets a warm instance.

What we’ll filter on

1. Signal quality: does the policy scale on something that tracks accelerator occupancy (in-flight concurrency, GPU memory) rather than arrival rate?
2. Burst absorption: what holds requests during the minutes between demand arriving and capacity arriving?
3. Time to usable capacity: how long from the scaling decision to an instance serving tokens?
4. Idle floor: can it shrink to zero overnight, and what does the first request after idle pay?
5. Interaction shape: does the caller hold a synchronous connection, or can it collect a result later?
6. Operational surface: how much of the scaling machinery does the team own and tune?

The landscape

The mechanism underneath almost every option here is Application Auto Scaling, which registers a scalable target (an endpoint variant’s desired instance count, or an inference component’s desired copy count) and drives it from a policy. The choices worth separating are which metric the policy watches and which kind of policy it is.

Target tracking on invocations per instance. The predefined metric SageMakerVariantInvocationsPerInstance is the default and the one the team already tried. One number, almost no configuration, and a good fit for a fleet where every request costs about the same. It counts arrivals, so it is blind to the difference between a morning of short lookups and a morning of thread summarisations. On generative traffic it under-provisions when answers are long and over-provisions when they are short, and the failure is quiet because the metric looks healthy either way.

Target tracking on concurrency. SageMaker publishes ConcurrentRequestsPerModel at high resolution, and the matching predefined scaling metric, SageMakerVariantConcurrentRequestsPerModelHighResolution, tracks in-flight requests rather than arrivals. That is the number the accelerators are actually holding, so it moves with generation length without being told about it. The high-resolution publication also strips a chunk of the detection lag out of the front of the scale-out chain. On an endpoint using inference components the equivalent is SageMakerInferenceComponentConcurrentRequestsPerCopyHighResolution against copy count. Of the available auto-scaling configurations this is the one to start from on a generative workload.

Step scaling. Instead of steering toward a target, step scaling puts CloudWatch alarm bands around the metric and adds a defined number of instances per band, so a large breach produces a large response. Target tracking deliberately smooths, which is right for drift and wrong for a wall of traffic. Step scaling as a second policy on a high-breach alarm gives a fast lane for the burst without giving up the steady-state behaviour of the target-tracking policy.

Scheduled scaling. A scheduled action sets minimum capacity on a cron, so the fleet is already at size when the traffic arrives rather than reacting to it. This is the direct answer to scale-out lag, and it applies whenever the shape is known and stable, which a weekday-morning internal tool absolutely is. The cost is paying for warm instances through the shoulder before the peak, which is cheap compared with paying for them all night.

Inference components with scale to zero. An inference component holds one model plus its declared resource requirements, and copies of it scale independently of the endpoint, down to zero copies. Managed instance scaling adds and removes instances underneath as copies need them. Adding a copy to an instance that is already running is much faster than adding an instance, so a fleet with headroom can absorb a step up in seconds rather than minutes. This is also the shape to be in if a second or third model shows up behind the same endpoint.

SageMaker Serverless Inference. Provisions on demand, scales to zero when idle, bills for the compute a request uses, and offers provisioned concurrency to hold a warm floor against cold starts. The traffic shape here is close to its ideal case. Two limits close it anyway: it runs on CPU with at most 6 GB of memory, so a GPU-served open-weight model cannot run on it at all. Worth naming precisely because the traffic pattern makes it look tempting.

SageMaker Asynchronous Inference. A queued endpoint on instance types you choose, GPUs included. Requests point at an S3 payload, land on an internal queue, get processed, and the result is written back to S3 with an optional SNS notification. It scales on backlog per instance and can sit at zero instances overnight, waking when the queue is not empty. The queue absorbs a burst by construction, which is the property the synchronous endpoint lacks. The trade is the interaction shape: the caller no longer holds a connection, so the client has to poll or be notified. The asynchronous pattern has its own shape, and it is a change to the application, not just to the scaling policy.

A queue and a worker fleet you own. SQS in front of workers on ECS or Fargate, scaling on queue depth or backlog per worker. Complete control over admission, priority, per-worker concurrency, and the scaling maths. In exchange the team owns the container, the health checks, the deployment, the accelerator scheduling, and every failure mode SageMaker hosting handles on its own. Reach for it when the managed shapes genuinely do not fit, not because the policy needs one more knob.

Batching strategies, of which there are two. Continuous (rolling) batching inside the serving container packs concurrent sequences into shared forward passes, so tokens per second per accelerator rises steeply until memory runs out. That moves the knee, which changes the capacity plan more than any scaling policy changes the fleet: a container without it can be running at a third of the throughput the same GPU would give with it. Separately, work with nobody waiting on it, nightly re-summarisation of yesterday’s tickets, backfills after a prompt change, belongs in a batch jobBatch inferenceSubmitting a bulk job of model calls to run asynchronously at a lower per-token price, trading immediacy for cost. rather than on the interactive endpoint, which takes that load off the morning entirely.

Not scaling anything. Moving to a managed foundation model on Bedrock on-demand removes the fleet and the policy, replacing the capacity decision with a quota. Provisioned throughput optimization is the Bedrock-side version of the same argument: reserve model unitsProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. to guarantee a floor for the sustained rate, and let on-demand absorb the shoulders, which is a sizing exercise of its own. What goes away with the fleet is the ability to tune the serving stack, and the reason the open weights were chosen in the first place.

Evaluation

Side by side

Option	Tracks occupancy	Absorbs a burst	Time to capacity	Can idle at zero	Synchronous caller	Team owns the machinery
Target tracking on invocations per instance	✗	✗	Minutes	✗	✓	Low
Target tracking on concurrency (high resolution)	✓	✗	Minutes	✗	✓	Low
Step scaling on a breach alarm	✓	✗	Minutes	✗	✓	Low
Scheduled scaling	n/a	✓ (pre-emptively)	Zero at peak	✓ (off-schedule)	✓	Low
Inference components, scale to zero	✓	✓ (spare copies)	Seconds to a warm instance	✓	✓	Medium
Serverless Inference	✓	✓	Cold start	✓	✓	Low
Asynchronous Inference	✓	✓ (queue)	Minutes from zero	✓	✗	Low
SQS with ECS or Fargate workers	✓	✓ (queue)	Minutes	✓	✗	High
Bedrock on-demand	n/a	✓ (quota)	None	✓	✓	None

Two things fall out of the table. No row absorbs a burst and serves a synchronous caller and scales on occupancy on its own, which is why the answer here is layered rather than picked. And the Serverless Inference row scores well on every column that this table measures, which is exactly why the 6 GB CPU-only limit has to be carried in the head rather than in a tick: a column-by-column comparison will recommend it right up until the model does not fit.

The solution

Take the load off first, because every later number gets easier. The nightly re-summarisation and the post-prompt-change backfills move to a batch job, so the interactive fleet stops competing with work nobody is waiting for. Then turn on continuous batching in the serving container and measure, on this model and this instance type, the sustainable tokens per second per instance and the concurrency at which first-token latency starts climbing without throughput following. That measurement is the capacity plan. Everything after it is arithmetic: peak arrival rate times mean duration gives peak concurrency, peak concurrency divided by the per-instance figure gives the instance count, and the same two numbers give the overnight floor.

The steady-state policy is target tracking on the high-resolution concurrency metric. Set the target below the measured knee rather than at it, so the fleet is still inside the useful part of the curve while new capacity is on its way. Minimum capacity stays above zero: overnight traffic is small but not absent, and the alternative is that the first person in at 06:40 pays a full model load. Maximum capacity is set from the measured peak with headroom, and the two cooldowns are deliberately asymmetric, short on scale-out and long on scale-in, so the fleet climbs quickly and comes down without flapping.

A scheduled action then raises minimum capacity to close to the peak count at 07:45 on weekdays and lowers it again mid-morning. This is what turns a losing race into a non-race: the burst is met by instances that were already warm, and the target-tracking policy handles the difference between the forecast and the day. Alongside it, a step-scaling policy on a high-breach concurrency alarm adds several instances at once when concurrency runs well past target, which is the case a smoothed target-tracking policy handles too politely: an incident, a launch, an all-hands that sends everyone to the assistant at once. Auto-scaling configurations that are optimized for GenAI traffic patterns are almost always this combination rather than any single policy, because the three parts answer three different questions: the schedule answers the known peak, target tracking answers the drift, and step scaling answers the surprise.

The long summarisations move to an asynchronous endpoint with its own scaling on backlog, and that separation does two useful things. It stops a 90-second generation sitting in the same concurrency budget as a 3-second lookup, which is what made the concurrency figure so noisy in the first place, and it lets that fleet sit at zero instances between jobs. Concurrent model invocation management then closes the loop at the container level: cap the maximum concurrent requests the serving container will admit, so it queues or sheds cleanly instead of thrashing key-value cache memory past the knee, and have callers retry with exponential backoff and jitter when they are turned away. A rejected request handled well is a much better outcome than an accepted one that times out.

What to watch

Utilization monitoring on a generative endpoint means a specific short list, and CPU is not on it. Watch GPUUtilization and GPUMemoryUtilization per instance, because memory is what runs out and it runs out before the accelerator is busy. Watch ConcurrentRequestsPerModel against the scaling target, which shows whether the policy is steering or chasing. Watch InvocationsPerInstance alongside it, not as a scaling signal but as the divisor that tells you whether a change in concurrency came from more requests or from longer answers. Split ModelLatency from OverheadLatency so a slow model and a slow endpoint are distinguishable, and track first-token latency separately from total, because a streaming client feels the first and not the second. Watch ModelSetupTime wherever anything scales from zero, since it is the number the whole scale-out lag argument rests on.

Then add the one metric nobody gets for free: output tokens served per instance-hour, published as a custom metric. Invocations per instance says how many requests the fleet took. Tokens per instance-hour, held up against the tokens per instance-hour the measurement said the fleet could sustain, says how much of the capacity being paid for is doing work. On this endpoint that ratio started somewhere around a fifth, and it is the number that makes the case for the schedule and the floor without anyone needing to read a scaling policy. It also degrades gracefully into a forecast: if the assistant’s user base doubles, the peak concurrency doubles, and the instance count follows from a number that was measured rather than guessed.

Worked example

A Tuesday, with the layered configuration in place.

At 07:45 the scheduled action lifts minimum capacity from 1 to 5. Five instances are warm and idle by 07:53, which costs roughly twenty minutes of instances nobody is using yet.

At 08:12 the first wave lands. Concurrency climbs past the target within two minutes and target tracking asks for a sixth instance, which is serving by 08:22. Nothing is queued in the meantime because the five scheduled instances were sized for most of the peak.

At 08:34 an outage notice goes out and the assistant takes a spike three times its normal peak. Concurrency runs well past target and trips the step-scaling alarm’s upper band, which adds three instances in one action rather than one at a time. The container’s admission cap holds the excess at the door with a retryable response for the four minutes before those instances are live, and the clients back off and come back rather than timing out on a half-open connection.

At 09:35 demand falls away. The long scale-in cooldown means the fleet stays at nine until 10:05 and then steps down gradually, which costs half an hour of instances and avoids paying for a second climb if the afternoon does something unexpected. At 10:15 the scheduled action drops minimum capacity back to 1, and by 11:00 the endpoint is running two instances against light traffic.

Overnight the endpoint sits at one instance rather than six. The asynchronous summarisation endpoint sits at zero and wakes on backlog when someone queues a job. The day bills around 55 instance-hours rather than 144, and the morning is faster than it was at 144, because the capacity is now in the right place at the right time rather than everywhere all the time.

What’s worth remembering

1. An LLM request holds an accelerator for seconds to minutes and its duration varies with output length, so in-flight concurrency, not invocations per instance, is the signal that tracks fleet load: concurrency equals arrival rate multiplied by mean duration.
2. Model load time means reactive scale-out arrives after the burst, so capacity has to be scheduled ahead of a known shape or the burst has to be queued, with target tracking covering the drift between them.
3. Accelerator memory is the binding resource and the concurrency knee is a measured number: past it, throughput per instance flattens while first-token latency climbs.
4. Continuous batching inside the serving container moves that knee further than any scaling policy moves the fleet, so measure with it switched on before sizing anything.
5. Scale out quickly and scale in slowly, and hold a floor above zero wherever the first request of the morning cannot afford to pay for a cold model load.
6. Serverless Inference runs on CPU with at most 6 GB of memory, so however well its scale-to-zero fits the traffic shape, it is not available to a GPU-served open-weight model.