The situation
A product team runs three GenAI workloads that all got built on whatever inference path was closest to hand, and the bill is now a mess of half-idle endpoints and throttling errors that nobody can explain.
The first is a support-reply assistant embedded in the agent console. It calls a Bedrock foundation model, sees steady traffic during business hours, roughly 15 to 40 requests per second, and needs a first token back fast because a human is waiting. The second is a nightly enrichment job: 4 million historical tickets get summarised and classified once, offline, with nothing waiting on the result before morning. The third is a fine-tuned open-weight model the data-science team trained on the company’s own taxonomy; it powers an internal triage tool that gets hammered for twenty minutes after each standup and then sees almost nothing for hours.
Three workloads, three completely different shapes. Each needs a serving option, and one option will not cover all three.
What actually matters
Inference cost comes from one of two places: capacity you hold and do not use, or capacity you do not hold when a request arrives. Every serving option on AWS answers the same pair of questions differently. Who holds the capacity, and when does the bill start? Match the option to the workload and you pay close to what you use. Mismatch it and you run an idle endpoint all day, or absorb throttling at peak.
The first axis is latency sensitivity. If a human is waiting on the first token, cold starts and queue time are unacceptable, so you hold warm capacity to avoid them. If the result is read minutes or hours later, you can give up latency instead, and giving it up is where most of the saving comes from.
The second is traffic shape: steady, spiky, or offline. Steady traffic calls for persistent capacity sized to the load. Spiky, intermittent traffic needs something that scales to zero between bursts. Offline, run-it-all-at-once traffic needs a batch mechanism that starts, processes the dataset, and shuts down, leaving nothing running.
The third is throughput guarantees. On-demand serving draws on a shared pool under account-level quotas, and under contention your calls are throttled. When a workload needs a floor of guaranteed throughput, or serves a model with no on-demand path, you reserve capacity and pay hourly whether you use it or not.
The fourth is the cost model itself: per-token, per-hour, or per-job. Per-token has no floor and tracks use. Once volume is both high and predictable, reserved per-hour capacity is cheaper. Per-job batch costs least per unit of work, and exists only for latency-tolerant workloads.
The last axis splits the options in half. Is the model a Bedrock-managed foundation model, or a self-hosted open-weight or custom one? Bedrock serves the managed FMs, plus custom and imported models under the conditions below. Anything you brought yourself, an open-weight checkpoint you fine-tuned or a bespoke architecture, runs on SageMaker hosting. That one fact splits the decision before any other axis applies.
What we’ll filter on
- Model provenance: a Bedrock-managed foundation model, or a self-hosted / custom model?
- Latency sensitivity: is a human (or a synchronous caller) waiting on the response?
- Traffic shape: steady, spiky and intermittent, or offline batch?
- Throughput guarantee: best-effort shared quota, or a reserved floor?
- Cost model that fits: per-token, per-hour reserved, or per-job?
The landscape
Bedrock on-demand. Pay per input and output token with no commitment and nothing to provision. You call InvokeModel or Converse, and Bedrock serves the request from a shared pool. This is the default for foundation-model workloads, and the starting point for almost anything interactive with variable volume. Throughput is governed by account-level service quotas, expressed as requests and tokens per minute per model. A busy workload hits ThrottlingException under contention, which is part of why cross-Region inference exists.
Bedrock Provisioned Throughput. Reserve capacity in model units, each delivering a set throughput for one named model, billed per hour. The term is your choice: no commitment, one month, or six months, with the longer terms discounted. Two reasons to reach for it. You need a throughput floor that account quotas will not give you, or you are serving a customised model with no on-demand path. Customising a model used to force the reservation outright; on-demand custom model deployments now cover a short list of bases (Amazon Nova Micro, Lite and Pro, Nova 2 Lite, and Llama 3.3 70B Instruct), customised on or after 16 July 2025, in two Regions. A model brought in through Custom Model Import is different again. Bedrock sizes it in custom model units, adds and removes model copies as demand changes, and bills in five-minute windows from the first successful inference call, so it needs no reservation. Reserved units bill whether or not traffic fills them, so they suit volume that is high and steady.
Bedrock batch inference. Submit a large set of records as a single asynchronous job (CreateModelInvocationJob), pointing at input in S3 and collecting output from S3 when it finishes. AWS prices batch at 50% below on-demand for the models that offer it. What you give up is interactivity: the job is queued and completes on its own schedule, so it suits work with nothing waiting on it. Two other limits matter. Each record is processed independently, so tool calling and structured output are unavailable, and provisioned or imported models cannot run batch jobs at all.
Cross-Region inference. A routing construct rather than a serving mode. An inference profile spreads invocations across the Regions of a geography, or globally, which raises effective throughput and reduces throttling. There is no additional routing charge, and a global profile is priced about 10% below a geographic one. Inference profiles do not work with Provisioned Throughput, so this modifies on-demand and nothing else.
When the workload isn’t a Bedrock FM at all, you’re on SageMaker hosting, which offers four serving shapes:
SageMaker real-time endpoints. A persistent HTTPS endpoint backed by one or more always-on instances, autoscaling on load. Lowest and most consistent latency, and the choice for steady, latency-sensitive traffic. Payloads run to 25 MB, with 60 seconds of processing for a regular response and 8 minutes for a streamed one. The instances bill around the clock, idle hours included, unless you host through inference components and set the variant’s managed instance scaling to a minimum of zero. An endpoint sitting at zero instances answers nothing until it provisions one, which takes several minutes.
SageMaker Serverless Inference. An endpoint that provisions compute per request and scales to zero when idle, billing the compute a request consumes plus the data processed. The first call after an idle period waits on a cold start, which CloudWatch reports as OverheadLatency. Two limits rule it out of most generative work: payloads cap at 4 MB with 60 seconds of processing, and it runs on CPU with 1 to 6 GB of memory and no accelerator option. That leaves it a fit for small CPU models on spiky traffic, and no use at all for a fine-tuned LLM.
SageMaker Asynchronous Inference. A queued endpoint for large payloads or long processing times. You pass the payload inline up to 128,000 bytes, or point at an S3 object for anything bigger. SageMaker queues the request, processes it, writes the result to S3, and can notify you over SNS. It scales to zero when the queue is empty, and it runs on the instance type you choose, GPUs included. Payloads go up to 1 GB and processing up to an hour, though a request times out at 15 minutes unless you raise InvocationTimeoutSeconds, whose ceiling is 3600.
SageMaker Batch Transform. Offline scoring of an entire dataset with no persistent endpoint at all. Point a transform job at data in S3 and it starts instances, processes every record, writes results back to S3, and shuts the instances down. It handles datasets in the gigabytes and processing times measured in days, with each request payload capped at 100 MB. This is the SageMaker analogue of Bedrock batch inference, for self-hosted models. If the nightly job used a custom model instead of a Bedrock FM, this is where it would run.
Those four answer “what shape is the traffic”. A fifth question cuts across all of them: how many models are you serving? Once the answer is dozens or hundreds, an endpoint per model bills mostly idle instances, and SageMaker offers two ways to pack them onto shared infrastructure.
Inference components. The current and more flexible of the two, and the one to reach for on generative workloads. An inference component is a hosting object holding one model plus its resource requirements, and you deploy several of them to one endpoint. Each declares what it needs (NumberOfCpuCoresRequired, MinMemoryRequiredInMb, NumberOfAcceleratorDevicesRequired) and how many copies to run. Each scales independently, down to zero copies so another component can scale up in its place. Because hosting is decoupled from the endpoint, models can be added, removed, and updated one at a time without touching the others. That per-model resource allocation is what fits a set of differently-sized models onto shared GPUs, which is the situation you get with several fine-tuned LLMs.
Multi-model endpoints. The older pattern, and still the right one for a large number of similar models. All of them share one serving container and one fleet. SageMaker downloads each model on first invocation, loads it into container memory, and unloads the least-used models when memory runs short. Adding a model means uploading it to S3 and invoking it, with no endpoint update and no code change, which is what makes hosting thousands of them practical. The constraints come with it: the models must share an ML framework and container, the first call to a cold model waits for the download and load, and the pattern works best when models are similar in size and latency. AWS recommends a dedicated endpoint for any model with materially higher throughput or latency requirements than its neighbours.
Multi-container endpoints are the third variation, hosting a handful of distinct containers behind one endpoint, invoked directly or chained as a serial inference pipeline. Reach for them when the models genuinely need different runtimes rather than when there are simply a lot of them.
Evaluation
Side by side
| Option | Model type | Latency fit | Traffic shape | Scales to zero | Cost model |
|---|---|---|---|---|---|
| Bedrock on-demand | Managed FM | ✓ interactive | Variable / spiky | ✓ (no floor) | Per-token |
| Bedrock Provisioned Throughput | Managed / custom FM | ✓ interactive | Steady, high volume | ✗ | Per-hour, per model unit |
| Bedrock batch inference | Managed FM | ✗ offline | Offline batch | ✓ (per-job) | Per-job, ~half on-demand |
| SageMaker real-time | Self-hosted | ✓ interactive | Steady | ✓ via inference components | Per-instance-hour |
| SageMaker Serverless | Self-hosted, CPU only | ✓ (cold starts) | Spiky / intermittent | ✓ | Per-request compute |
| SageMaker Async | Self-hosted | ✗ synchronous | Long / heavy payloads | ✓ | Per-instance-hour (queued) |
| SageMaker Batch Transform | Self-hosted | ✗ offline | Offline batch | ✓ (per-job) | Per-job instance-hours |
| SageMaker inference components | Self-hosted | ✓ interactive | Many models, mixed sizes | ✓ (per component) | Per-instance-hour, shared |
| SageMaker multi-model endpoint | Self-hosted | ✓ (cold model penalty) | Many similar models, long tail | ✗ | Per-instance-hour, shared |
The two halves of the table never compete directly; model provenance picks the half, then latency and traffic shape pick the row. The last two rows are the exception, because they answer a question about model count rather than traffic shape: reach for them when the alternative is standing up an endpoint per model.
The numbers that decide it
On the SageMaker half, a hard limit usually settles the choice before preference does. A payload size and a processing time in the requirements eliminate most of the rows on their own, before traffic shape is considered at all.
| SageMaker option | Max payload | Max processing time | GPU | Scales to zero |
|---|---|---|---|---|
| Real-time endpoint | 25 MB | 60 s (8 min streaming) | ✓ | ✓ with inference components |
| Serverless Inference | 4 MB | 60 s | ✗ (CPU only, 1-6 GB RAM) | ✓ |
| Asynchronous Inference | 1 GB | 60 min (15 min default) | ✓ | ✓ |
| Batch Transform | GB-scale datasets | Days | ✓ | ✓ (per-job) |
Read it as a filter. A requirement for GPU inference removes Serverless outright, which takes out most generative work. A payload over 25 MB removes real-time. A response needed within minutes rather than overnight removes Batch Transform. What survives a large-payload, GPU-bound, minutes-not-hours requirement is Asynchronous Inference. Remember the 15-minute default timeout there: it is the number that catches a job that used to finish in ten minutes and has since grown.
The solution
The support assistant lands on Bedrock on-demand. It’s a managed foundation model, a human is waiting, and traffic is variable within the business day. On-demand bills per token, with no idle cost and no capacity to plan. Quotas are the thing to watch. At 15 to 40 requests per second the workload can reach the per-minute request and token limits, so track ThrottlingException rates and request quota increases where the ceiling is real. A cross-Region inference profile raises effective throughput without a reservation, so try that before Provisioned Throughput. Reserved model units come later, once volume is steady enough that the per-hour rate beats the per-token one, or once an SLA needs a guaranteed floor.
The nightly enrichment job lands on Bedrock batch inference. Four million tickets, offline, nothing waiting: this is latency-tolerant work. Running it through synchronous InvokeModel calls would cost twice as much per token and contend with the interactive workload for the same quota. Batch inference reads the records from S3, runs them as one managed asynchronous job at half the on-demand rate, and writes results back to S3. Size the input, start the job after hours, collect the output by morning. If this job used a self-hosted model instead, the equivalent is a SageMaker Batch Transform job.
The triage tool lands on a SageMaker real-time endpoint that scales to zero. It’s a fine-tuned open-weight model, so Bedrock’s managed-FM paths are out and it belongs on SageMaker hosting. The traffic then suggests Serverless Inference, and the model rules it out: Serverless runs on CPU with at most 6 GB of memory, and a fine-tuned LLM fits in neither. Host it as an inference component instead, on a GPU endpoint whose managed instance scaling has a minimum of zero. It bills nothing while it sits at zero, and provisioning takes several minutes on the first request afterwards. Because standup happens at a known time, a scheduled scaling action warms the endpoint just before the burst. Asynchronous Inference is the alternative if the callers can collect results from S3 rather than hold a connection open.
There’s a subtlety worth stating plainly. A fine-tuned model can end up on either half of the tree depending on how it was made. Fine-tune a model into Bedrock and it serves on Bedrock, though whether on-demand is available depends on the base model and the customisation method. Fine-tune an open-weight checkpoint yourself and it serves on SageMaker hosting, or comes back through Custom Model Import. Same phrase, “we fine-tuned a model,” two entirely different serving decisions, so establish which one before picking anything.
Worked example
Put numbers on the three workloads and the shapes separate cleanly.
The support assistant runs about 20 requests per second for eight business hours, call it 576,000 requests a day, each a few hundred tokens in and out. On-demand per-token pricing tracks that usage and falls to nothing overnight, with no idle floor and nothing to operate but quota headroom. Provisioned Throughput would mean reserved model units billing 24 hours a day to cover an 8-hour load. That is cheaper only when daytime volume keeps the units saturated.
The enrichment job processes 4 million records once a night. As synchronous on-demand calls it pays the full per-token rate and contends with the assistant’s quota. As a batch job it pays half and runs in its own lane. That is a 50% reduction on 4 million records of input and output tokens, every night, in return for a result that lands by morning instead of instantly.
The triage tool sees maybe 400 requests in a twenty-minute window and a trickle afterwards. An always-on GPU instance sized for the burst bills 24 hours to serve well under an hour of real work. An inference component with a minimum of zero copies holds no instance between bursts. A scheduled scaling action just before standup brings a copy up ahead of the first request, keeping the several-minute provisioning time off the critical path.
Same team, three workloads, three different serving options, and each choice falls out of the traffic shape and the model’s provenance rather than any property of the model itself.
What’s worth remembering
- Model provenance splits the decision before anything else: Bedrock-managed foundation models go on Bedrock’s serving paths, and self-hosted or open-weight models you trained go on SageMaker hosting.
- “We fine-tuned a model” does not settle that split, because a model customised inside Bedrock stays on Bedrock, where on-demand serving covers only a short list of base models and everything else needs Provisioned Throughput.
- Bedrock on-demand is the default for interactive FM workloads: per-token, no commitment, no idle cost, with throughput bounded by account quotas, so track throttling and use a cross-Region inference profile before reserving model units.
- Bedrock batch inference runs offline record sets as an S3-to-S3 job at 50% below the on-demand rate, and gives up interactivity, tool calling and structured output in return.
- On the SageMaker half the hard limits settle it before preference does: real-time caps at 25 MB and 60 seconds (8 minutes streaming), Serverless at 4 MB and 60 seconds on CPU with up to 6 GB of memory, and Asynchronous at 1 GB and 60 minutes with a 15-minute default timeout.
- Serverless Inference has no GPU option, so spiky traffic to a self-hosted LLM belongs on a real-time endpoint hosting inference components that scale to zero copies, or on Asynchronous Inference, both of which run on the instance type you choose.