Choosing How a Model Serves Its Predictions

The situation

A subscription produce-box business has one trained model in production. It takes a basket, a payment method and a dozen account features, and returns a fraud risk score between 0 and 1. It was trained in Amazon SageMaker AI, it is a few hundred megabytes on disk, and it runs perfectly well on CPU. Nobody is arguing about the model.

Four teams want to call it, or something like it, and all four have raised a ticket asking for an endpoint.

Checkout needs a score before the payment is authorised. The page cannot commit the transaction until the score comes back, so the budget is under 200ms at the ninety-ninth percentile. Traffic runs between forty and ninety requests a second through the day, dips overnight and never quite reaches zero.

Risk needs every subscriber re-scored once a night. That is 2.4 million rows exported from the subscriptions database, scored, and written back for the morning review queue. The job starts once the day’s transactions have settled and has to be finished by 6am. Nobody is sitting waiting on any individual row.

The fraud analysts want a screen: paste a basket, get back the score and the features that pushed it up. Sixty or so lookups a day, clustered in office hours, nothing overnight and nothing at weekends. An analyst will wait half a minute for an answer without complaining.

Onboarding wants something different again. They have a document model that reads scanned supplier certificates. One submission is a bundle of scans up to 400MB, and one bundle takes several minutes to work through. They arrive a few dozen times a week at no predictable hour, and the person who uploaded one goes back to their inbox and collects the result later.

Four asks, one word, four different jobs.

What actually matters

Start with the word. Inferencing is what a model does after it has been trained: it is handed data it has not seen, and it produces an output. Training happens occasionally and costs a lot; inferencing happens constantly and is most of what the business pays for month to month. Batch, real-time, asynchronous and serverless all perform the same inferencing with the same trained model. What differs is how the request reaches the model, and what infrastructure is standing by when it arrives.

The first property to settle is whether anyone is waiting for the answer inside the same request. A checkout page holds a connection open and cannot render until the score arrives, so every millisecond is on somebody’s clock. The nightly re-score has nothing on the other end of it. Rows are read from one place and written to another, and the only deadline is the 6am one on the job as a whole. Those two are not the same requirement expressed at different sizes. One is a latency budget, the other is a completion deadline, and they are satisfied by different machinery.

Second is the shape of the traffic and, specifically, the length of the gaps. Checkout traffic never stops, so an instance sitting there is an instance doing work. The analyst screen goes sixteen hours a day without a single call, and something has to happen during those sixteen hours. Either the instance stays up and is paid for while it waits, or it goes away and something brings it back when the next request arrives. Bringing it back takes time. That waiting time on the first request after an idle period is a cold start, and whether it is acceptable is a product question, not an infrastructure one. It is fatal at checkout and invisible on a screen where the analyst expects to wait anyway.

Third is size and duration: how big is one request, and how long does the model take on it. Request-and-response inferencing has ceilings on both. An HTTP connection held open for ten minutes gets dropped by something in the middle. A payload measured in megabytes and a response measured in tens of seconds fit that shape. A 400MB bundle taking several minutes does not, and no amount of instance sizing changes that, because the constraint is the request, not the compute.

Fourth is what you are willing to pay for idleness. An always-on endpoint bills for the instance, not for the predictions, so a service handling sixty requests a day pays the same hourly rate as one handling six million. That may be entirely reasonable. It is a decision worth making deliberately rather than inheriting from whoever set up the first endpoint.

These four properties do not sort the options into better and worse. They sort them into jobs. This scenario is about a model the team trained themselves; the same four properties decide the serving surface when the model is a foundation model instead, which the professional-level material works through in more depth.

What we’ll filter on

1. Does a caller wait for the answer inside the same request, or does it collect the result later?
2. What is the latency budget, if there is one at all?
3. How steady is the traffic, and how long are the idle gaps between requests?
4. How large is one payload, and how long does the model take on it?
5. Does the option scale to zero, and what does it cost while nothing is happening?
6. Where do the input and the output live: in the request body, or in Amazon S3?

The landscape

Amazon SageMaker AI offers four ways to serve a trained model, and each one exists because of a job the others do badly.

Real-time inference on an endpoint

You deploy the model to an endpoint backed by one or more instances that stay running. A caller sends a request and gets the prediction back in the response, in a few milliseconds to a few hundred. Autoscaling adds and removes instances as traffic changes, within a minimum you set, and that minimum is at least one for the endpoint to answer at all.

The ceilings come from the request-and-response shape: a payload of a few megabytes, and a response inside about a minute. You choose the instance type, including a GPU one if the model needs it, and you are billed for those instances for as long as the endpoint is up, whether or not anything calls it.

Serverless inference

The same request-and-response shape with no instances to choose. You set a memory size (single-digit gigabytes) and a maximum concurrency, and SageMaker AI runs the container when a request arrives and stops running it when the traffic stops. Billing is by the compute time a request actually consumes plus the data processed, so an idle hour costs nothing.

The trade is the cold start. After a quiet period, the first request has to wait while the container is brought up and the model is loaded, which can take from a second or two to considerably longer for a large model. There is no GPU option, and the memory ceiling is real. A large deep-learning model is often out of range. Provisioned concurrency will keep capacity warm, at which point you are paying for readiness again.

Asynchronous inference

A managed queue sits in front of the endpoint. The caller does not send the data in the request; it uploads the input to S3 and sends the location. SageMaker AI returns an output location immediately, processes the request when it reaches the front of the queue, writes the result to S3 and, if you have configured it, publishes a notification when it is done.

Breaking the connection lifts both ceilings at once. Payloads run up to a gigabyte and processing time up to about an hour, well past anything a synchronous call would survive. The endpoint can also autoscale down to zero instances when the queue drains, so a quiet weekend costs nothing, at the cost of a start-up wait on the first item to arrive afterwards.

Batch transform

No endpoint at all. You point a batch transform job at a dataset in S3, SageMaker AI provisions instances, splits the data across them, runs every record through the model, writes the predictions back to S3 and tears the instances down. You are billed for the duration of the job.

There is nothing left running when it finishes and nothing to call between jobs. The unit of work is a dataset rather than a request, which is why the job is measured by when it completes rather than by how quickly any single row was scored.

Evaluation

Side by side

	Caller waits in-request	Latency of one prediction	Payload ceiling	Duration ceiling	Scales to zero	Paid while idle
Real-time endpoint	✓	milliseconds, consistent	a few MB	about a minute	✗	✓
Serverless inference	✓	milliseconds, plus cold starts	a few MB	about a minute	✓	✗
Asynchronous inference	✗	seconds to minutes	up to ~1GB	up to ~1 hour	✓	✗
Batch transform	✗	not meaningful per record	the dataset	length of the job	✓	✗

The last row is the one people misread. A batch transform job has no latency figure to compare with the others, because there is no caller and no clock on any individual prediction. Putting it in the same column as the other three invites the conclusion that it is a slower, cheaper version of an endpoint. It is a different job entirely.

Which way the decision runs

The gates run in that order because each one answers a question the next cannot. Whether the work arrives as a dataset or as requests decides whether an endpoint is involved at all. Whether a caller waits decides whether the size and duration ceilings apply. Only after both of those does traffic shape get a vote, and traffic shape is where most people start.

The solution

Checkout goes on a real-time endpoint. It is the one ask with a latency budget attached to a customer-facing page, the payload is a few kilobytes of basket and account features, and traffic never falls to zero, so idle capacity is not something anyone is paying for by accident. Autoscale on invocations per instance, with a floor of at least two instances across availability zones. Keep that floor high enough that a morning ramp does not spend its first minutes waiting for capacity.

The nightly re-score goes to batch transform. There is no caller, the input is already a file (or can be, once somebody owns the nightly export from the subscriptions database into S3), and the output is another file that the morning review queue reads. Splitting 2.4 million records across a handful of instances for forty minutes and then shutting them down costs a fraction of what an endpoint sized for that burst would, and it cannot disturb checkout, because it shares nothing with it.

The analyst screen goes on serverless inference. Sixty invocations a day against an always-on instance means paying twenty-four hours of instance time for a few minutes of work, and the sixteen-hour overnight gap is exactly the idle that serverless is built to stop charging for. Cold starts are the trade. An analyst who already expects a pause when they hit the button will not notice a few extra seconds on the first lookup after lunch. Check the model loads inside the memory ceiling and does not want a GPU before committing.

The scanned certificate bundles go to asynchronous inference. A 400MB payload and several minutes of processing are both outside what a synchronous call can carry, and the workflow already matches the queue: upload, get an output location, come back later. Configure the completion notification so the onboarding tool can tell the person their bundle is ready rather than polling for it, and let the endpoint scale to zero on the quiet days.

Two things are worth watching after that. The first is the assumption that asynchronous inference is the option for intermittent traffic. It does scale to zero. The reason to reach for it is size and duration, and a small, fast, intermittent workload gets simpler treatment from serverless. The second is the temptation to consolidate. Four serving surfaces looks like three too many until the nightly job saturates the checkout endpoint at 3am and a customer’s payment times out. The separation is what keeps that from becoming an outage.

Worked example

Take the tempting shortcut and run the nightly re-score through the checkout endpoint. The model takes about 8ms per record, but the round trip through an endpoint is closer to 40ms once serialisation and network are counted. Push 2.4 million records through it one at a time and that is 96,000 seconds, or roughly twenty-six hours, on a single connection. Parallelise to fifty concurrent callers and it lands near half an hour. By then the endpoint is running flat out and every checkout request is queueing behind the job.

Batch transform does not have that arithmetic. The job reads the dataset from S3, splits it across five instances, and feeds records to the container in mini-batches rather than one HTTP request at a time. The predictions are written back to S3. The per-record overhead is close to the model’s own 8ms rather than 40ms, no connection is held open for any of it, and the instances exist only for the length of the run.

What’s worth remembering

1. Inferencing is using a trained model on new data, and batch, real-time, asynchronous and serverless are four ways of delivering it rather than four speeds of the same one.
2. Real-time endpoints keep instances running for consistent low latency and bill for them whether or not requests arrive.
3. Serverless inference scales to zero and pays for it with cold starts on the first request after idle, with a memory ceiling and no GPU.
4. Asynchronous inference is the answer to large payloads and long processing, not to intermittent traffic; it takes an S3 location, returns one, and can scale to zero as well.
5. Batch transform is a different shape of job rather than a cheaper endpoint: a dataset in S3, no persistent infrastructure, and a completion deadline instead of a latency budget.
6. Sort the ask by whether a caller waits, then by payload size and duration, then by traffic shape; starting at traffic shape argues about instance types before anyone has established that an endpoint is needed.