The situation
A logistics company has three model-backed features converging on the same release train. The customer-facing assistant calls a hosted model that AWS operates; its traffic is spiky and daytime-shaped, near zero overnight. A document classifier is midway through a training run on the team’s own labelled data, due in about three weeks. And the research group has a specialised extraction model they trained on their own hardware, with the weights sitting in an S3 bucket waiting for somebody to decide what happens next.
Finance has asked for a twelve-month serving forecast covering all three. It is a reasonable request and nobody can answer it. The three features do not merely cost different amounts; they are metered on different things. One accrues charges only when a request arrives. One will accrue them by the hour whether or not anybody uses it. The third has not been decided yet, and the team has been assuming that decision is theirs to make on price.
That last assumption is the one that hurts. Two of the three have already had their billing shape settled by choices made weeks ago, before anybody drew up a forecast, and reversing either means retraining rather than reconfiguring.
What actually matters
Start with where the options come from, because that is the part teams expect to control and mostly cannot. The set of ways a model can be served is fixed by its provenance: whether the weights are ones the platform hosts for everybody, ones you adapted through the platform’s own training, or ones you produced elsewhere and brought with you. Each of those origins opens a different set of surfaces, and within the middle one the specific model you started from narrows it further. By the time there is a bill to look at, the option set is already decided.
The second is that the billing units differ in kind rather than in rate. One surface meters the tokens a request consumes. Another meters reserved capacity by the hour, regardless of whether requests arrive. Another meters the minutes during which capacity is live. Another meters the instances you keep running. You cannot compare these by looking at their prices, because they are prices of different things. A comparison only becomes meaningful once you supply a traffic shape, and the same two surfaces will swap places depending on whether the workload is a steady grind or ninety busy minutes a day.
The third is what happens when nothing is happening. A unit that meters time keeps metering overnight, at weekends, and through the quiet fortnight after a launch. A unit that meters consumption stops. Between them sits the surface that stops charging after a period of idleness but makes the next caller wait while capacity comes back, which is a latency cost paid in exchange for a billing one. Whether that trade is acceptable is a question about the workload’s tolerance, not about its budget.
The fourth is commitment. Where capacity is reserved, it can usually be reserved for longer in exchange for a lower rate, which is a straightforward trade of price against flexibility and a poor bet on a workload whose volume nobody has measured yet. The discount is real and so is the lock-in, and a term chosen before the first month of production traffic is a guess, whatever the saving on paper says.
The last is that serving is not the only line on the invoice. Adapting a model costs something at training time, usually metered on the data processed and the number of passes over it. The resulting artefact then costs something every month it exists, whether or not it is serving, until somebody deletes it. Teams forecast the inference and are surprised by the rest, and the rest is the part that keeps accruing after a feature is quietly switched off.
What we’ll filter on
- Where did the weights come from, and does that origin leave more than one serving surface open?
- Does the bill follow the traffic, or does it follow the clock?
- Does it stop when the workload stops, and what does the first request after a quiet period cost in latency?
- What commitment does it ask for, and how reversible is that commitment?
- What accrues when nothing is being served: reserved capacity, idle instances, stored artefacts?
The landscape
On-demand invocation of a hosted model. Call the model, pay for the tokens the call consumed, input and output priced separately with output usually dearer. No capacity to reserve, no floor, nothing accruing between calls. This is the default surface for models AWS hosts, and it is the one every prototype starts on. Two variants sit alongside it: batch inference, which takes the same work at a discount in exchange for a service level measured in hours rather than seconds, and cross-region inference profiles, which spread load across regions at the same per-token rate to soften throttling. Fits spiky, interactive, and exploratory workloads, which is most of them.
Provisioned Throughput. Reserve capacity in Model unitThe billing block Provisioned Throughput is sold in – one unit delivers a fixed tokens-per-minute rate for a specific model., each delivering a defined throughput for one specific model, and pay by the hour for as long as the reservation exists. Available with no commitment at the highest rate, or for a one-month or six-month term at progressively lower ones. The eligibility rule is the part that catches people: AWS publishes a list of foundation model IDs that Provisioned Throughput can be purchased for, covering those base models and any model you customised from one through Bedrock’s own training. Nothing outside that list is eligible, whatever else you may have in the account. Fits steady high volume, a guaranteed throughput floor, and the customised models that have no other option.
On-demand serving of a customised model. Some bases, once customised through Bedrock’s training, can be deployed for on-demand inference and billed per token at the base model’s rates, with nothing reserved. The Nova family works this way, and so does Llama 3.3 70B. Others do not, and for those Provisioned Throughput is the only path, priced on the base model’s unit rate. Which side of that line a customisation falls on is a property of the base you picked at the start, so it is a question to settle before training rather than after. The route you take to customise does not change the answer; the base does.
Custom Model Import. Bring weights trained elsewhere into Bedrock’s managed serving and call them through the same API surface as anything else. Billing is by the Custom Model Unit minute of active use, with a minimum billable window, plus a monthly storage charge per imported model. Capacity scales to zero after a period of idleness and the next call pays a cold start of tens of seconds. Imported models are not on the Provisioned Throughput eligibility list, so this per-minute meter is the billing shape, not one option among several. Fits sporadic or clinic-shaped traffic on weights you own, and supports a limited set of architectures.
Self-hosting on SageMaker. Run the weights on infrastructure you manage. A real-time endpoint bills instance-hours for as long as it exists, giving predictable latency and total control at the cost of paying through every quiet hour. Serverless inference bills per invocation and scales to zero, at the price of cold starts that can be slow for large models. Asynchronous inference sits between them for work that tolerates queuing. Fits architectures Bedrock will not accept, deployment control Bedrock does not expose, and teams who would rather own the operational surface than the constraint list.
Evaluation
Side by side
| Surface | Applies to | Billing unit | Stops when idle | Commitment |
|---|---|---|---|---|
| On-demand | Hosted models | Per token | ✓ | None |
| Batch | Hosted models | Per token, discounted | ✓ | None |
| Provisioned Throughput | Listed base models, and Bedrock customisations of them | Per model-unit hour | ✗ | None, 1 month, or 6 months |
| Custom model deployment | Customisations of bases that support it | Per token | ✓ | None |
| Custom Model Import | Weights trained elsewhere | Per unit-minute active | ✓ (cold start after) | None |
| SageMaker real-time | Anything | Instance-hours | ✗ | None (or Savings Plans) |
| SageMaker serverless | Anything | Per invocation | ✓ (cold start after) | None |
Which surfaces a model can reach
Reading the diagram against the three features: the assistant has the widest choice and should keep it, the classifier’s choice was made when somebody picked its base, and the extraction model has one managed option and one self-managed one, with reservation available for neither.
The solution
The assistant stays on demand. Spiky daytime traffic against a hosted model is the case per-token billing exists for: nothing accrues overnight, nothing is committed, and the bill tracks the feature’s actual use closely enough that finance can forecast it from a request count. Provisioned Throughput would be a reasonable thing to revisit only if the traffic flattened into a steady, high, predictable grind, or if on-demand throttling started costing user-visible latency, which a guaranteed floor would fix. Neither is true, so the work here is measurement rather than architecture: know the token volume per request and per day, and the forecast follows.
The classifier’s serving shape is already decided and the team should find out which way before the training run finishes rather than after. If its base is one that supports on-demand custom serving, the classifier deploys and bills per token at base rates, and it behaves like the assistant for forecasting purposes. If not, Provisioned Throughput is the only path, and the forecast changes character completely: a reserved unit bills every hour it exists, so a classifier processing a few thousand documents in a daily batch would spend most of the month paying for capacity nobody is using. That is the case where the unit count and the term become the whole decision, and where a nightly batch window rather than a live endpoint may be the cheaper design. Either way it is worth confirming now, because if the answer is unwelcome the remedy is a different base and another training run.
The extraction model has the cleanest decision of the three, because reservation is not available to it at all. Weights trained outside Bedrock are not on the Provisioned Throughput eligibility list, so the real choice is managed import against self-hosting. If the architecture is one import supports, the per-minute meter suits research-shaped traffic well: capacity scales to zero between uses, and the standing cost is the monthly storage on the stored model rather than a running endpoint. The cold start on the first call after a quiet period is the thing to check against the feature’s latency budget. If the architecture is not supported, or the team needs deployment control that managed serving does not expose, SageMaker takes it at instance-hours, which means designing for the quiet hours explicitly rather than discovering them on the invoice.
Across all three, the charges that are not inference deserve a line of their own in the forecast. Customising bills for the data processed and the passes over it, once. The resulting artefact bills monthly for as long as it exists. An imported model bills monthly for its stored weights. None of these follow traffic, so none of them shrink when a feature turns out to be unpopular, and all of them keep accruing after it is switched off unless somebody deletes the artefact.
Worked example
Take the classifier at a realistic volume and price both branches, because the arithmetic is what makes the constraint feel concrete. Say it handles thirty thousand documents a month, arriving in a weekday overnight batch that takes about ninety minutes, and each document costs roughly two thousand input tokens and two hundred output.
On the per-token branch, the bill is a multiplication: sixty million input tokens and six million output tokens a month, at the base model’s published rates. Nothing else accrues. Double the document count and the bill doubles; halve it and it halves. Finance can forecast this from a document count alone, which is the property that makes it easy to defend.
On the reserved branch, the arithmetic changes shape. The unit count comes from the busiest minute of that ninety-minute window, not from the monthly total, because the reservation has to be large enough for the peak it must absorb. Once sized, that unit bills for all seven hundred and thirty hours in the month, of which about thirty are doing work. The other seven hundred are the cost of the constraint. At that ratio, the monthly bill barely moves whether the classifier processes thirty thousand documents or three hundred thousand, which is worth understanding before treating it as a disaster: the reserved branch is punishing at this volume and would become competitive if throughput rose by an order of magnitude. What makes it painful here is not the rate, it is the mismatch between a workload that runs ninety minutes a day and a meter that runs all day.
That comparison is why the base model chosen at the start of the training run is worth an hour of somebody’s attention. The two branches are not slightly different prices for the same thing; they are different relationships between usage and cost, and only one of them shrinks when the feature is quiet.
What’s worth remembering
- Provenance decides the option set: hosted weights, weights customised through Bedrock, and weights trained elsewhere each open a different set of serving surfaces, and the choice is made before there is a bill to look at.
- Provisioned Throughput eligibility is a published list of AWS-provided foundation models, covering those base models and Bedrock customisations of them; imported weights are not on it and cannot be reserved at any price.
- Whether a customised model can serve on demand is a property of the base it was built from, so confirm it before the training run rather than after.
- The units differ in kind, not rate: per token, per reserved unit-hour, per active minute, per instance-hour. A price comparison is meaningless without a traffic shape.
- Anything metered on time bills through the quiet hours, which is why a workload that runs ninety minutes a day is the worst possible fit for a reservation and a good fit for anything that scales to zero.
- Training charges once and stored artefacts charge monthly until deleted, so a switched-off feature keeps costing something until somebody removes the model.