The situation
An insurance broker runs a claims team of forty caseworkers. Every claim arrives with paperwork: medical reports, repair quotes, incident statements, letters from solicitors. A typical document runs twelve pages, around six thousand words, and a caseworker used to read all of it to find four facts.
In June the team put a Summarise button in the case-management screen. It sends the document to a foundation model on Amazon Bedrock and returns a four-hundred-word summary. Caseworkers press it about three thousand times a day, and roughly a third of the time they follow up with three more questions about the same document without leaving the screen. Separately, a nightly job summarises two thousand documents from the archive so older claims become searchable, and a one-off backlog of sixty thousand archived documents is queued for the same treatment.
The first full month’s Bedrock bill came to a little over USD$2,000 against a whiteboard estimate of USD$500. Nobody on the team can say which part of the workload produced it, because nobody can say what a single call costs. That is the first thing to fix. The second is deciding how each part of this workload should be paid for.
What actually matters
Start with the unit. AWS defines a token as a sequence of characters a model interprets or predicts as one unit of meaning. That can be a whole word, a fragment carrying grammatical sense such as “-ed”, a punctuation mark, or a common phrase. As a planning rule of thumb, a thousand tokens is roughly seven hundred and fifty words of ordinary English. Models read and generate in tokens, and Bedrock meters them. That is the token-based pricing model: you are charged for the input tokens you send and the output tokens the model generates. The invocation itself adds nothing to the meter, however long it takes on the clock. Two rates apply rather than one, and the output rate is usually several times the input rate, because the model reads a prompt in one pass and produces output one token at a time.
The second thing to understand is that every call is charged in full, every time. A foundation model holds nothing between calls, so anything the next answer depends on has to be sent again. The Summarise button sends the whole twelve-page document. A follow-up question sends that document again, plus the summary, plus the earlier questions and answers, plus the new question. By the third follow-up the team has paid for that document four times. Everything that occupies the context window is charged at the input rate on every call that carries it, so a long system prompt and a replayed conversation are billed again on every turn.
Output tokens do double duty: they set the bill and they set the wait. A model generates one token at a time, in order, so a nine-hundred-word answer takes about three times as long to arrive as a three-hundred-word one and costs about three times as much. Capping the length of the answer improves both numbers with one setting, which no other lever here does. Where a wait is unavoidable, streaming tokens back as they are generated changes how the wait feels without changing what it costs.
Then there is who is waiting. Responsiveness is a property of the work rather than of the model. A caseworker watching a spinner needs an answer in seconds; nothing downstream of the archive job changes if a document finishes at 5am instead of 2am. That difference is worth money, because Bedrock prices asynchronous batch work fifty per cent below on-demand for identical tokens.
The last set of concerns is where the capacity comes from. On-demand capacity is shared and governed by account quotas, so a burst past the limit is throttled. That is availability. Retries with backoff, and somewhere else to send the traffic, are the redundancy. Regional coverage is a harder constraint than teams expect. A given model is not offered in every AWS Region, so if this claims data may only sit in one country, the catalogue available to the workload is whatever that Region offers. A cross-Region inference profile routes one workload’s calls to other Regions to raise the ceiling under load. AWS adds no routing charge and prices each call by the Region it was made from. A geographic profile confines routing to one geography such as US or EU; a global profile routes to any commercial AWS Region and lists around ten per cent lower, so it suits data with no residency constraint.
What we’ll filter on
AWS names the cost trade-offs of its GenAI services as responsiveness, availability, redundancy, performance, regional coverage, token-based pricing, provisioned throughput, and custom models. Applied to this workload, that list becomes six things to compare the options on.
- Responsiveness: is a person waiting on this call, and what breaks if the answer arrives in four hours instead of four seconds?
- What the bill scales with: the tokens consumed, or the time capacity is held whether or not anything flows through it?
- The rate on identical tokens: does this option charge more, less, or the same for the same input and output?
- Availability and redundancy: what happens in the busiest hour, and is there anywhere else for the traffic to go?
- Regional coverage: is the model offered in the Regions this data is permitted to sit in?
- Commitment: how long are we tied in, and what does an idle hour cost?
The landscape
There are four ways to pay for inference on Bedrock, the last of which splits in two depending on where the weights came from, and a workload with several shapes of job in it will usually want more than one of them.
On-demand invocation
The default, and what the Summarise button is doing today. Call the model, get an answer back in the same request, pay the published per-token rate for that model in that Region. No commitment, nothing to provision, and an hour with no traffic costs nothing at all. Throughput sits under account quotas shared with everything else calling that model in that Region, so a burst past the limit returns a ThrottlingException and the application has to retry. Latency varies with how busy the shared pool is.
Batch inference
Write the requests into a file in Amazon S3, submit one job, and collect the results from S3 when it finishes. Bedrock charges fifty per cent less than on-demand for exactly the same tokens. There is no live response and no per-request latency worth quoting, because the unit of work is a job rather than a request. You set the job timeout anywhere from 24 to 168 hours, which tells you the shape of the guarantee: a window measured in days rather than a response measured in seconds. Batch covers a subset of the catalogue rather than every model in every Region, so check the supported list before planning around it. Serving a trained model on Amazon SageMaker AI draws the same line in its own vocabulary, between work somebody is waiting for and work nobody is.
Provisioned Throughput
Buy capacity rather than tokens. You purchase model units, each delivering a set number of input and output tokens per minute, at one of three commitment levels: none, one month, or six months. The longer the term, the lower the hourly price. The bill is then flat. It is the same number whether the units run hot all day or sit idle from midnight. The arithmetic works only when sustained volume keeps them busy for most of the hours being paid for. What comes with it is throughput and latency that do not move when the shared pool gets busy. Inference profiles do not support Provisioned Throughput, so this option and the cross-Region one above do not combine. The commitment maths, and the capacity tiers that have grown up alongside this option, are worked through at length in matching Bedrock pricing to a workload’s rhythm.
Custom models and their hosting
A custom model is either a base model you have changed or a set of weights you have brought with you, and the two are served on different meters. A Bedrock fine-tune reaches inference by one of two routes. Buy Provisioned Throughput for it, charged by the hour at the base model’s rate, or create a custom model deployment and invoke that on demand per token. The on-demand route is narrow. The model has to have been customised on or after 16 July 2025, and the supported bases are a short list: Amazon Nova Micro, Nova Lite, Nova Pro and Nova 2 Lite in US East (N. Virginia), and Meta Llama 3.3 70B Instruct in US West (Oregon). Fine-tune anything else and Provisioned Throughput is the only way to call it.
Weights trained outside Bedrock take a different route. Custom Model Import puts them behind the same API and serves them on demand, metered by the Custom Model Unit across five-minute billing windows that start at the first successful inference call. Bedrock removes copies that are not active, so an idle model is not billed, and a call arriving when no copy is running returns ModelNotReadyException. The SDK retries that with exponential backoff while a copy restores, and how long the restore takes depends on the model size and the on-demand fleet. Batch inference is not available for imported models. Training a fine-tune is charged once on the tokens processed, importing weights is not charged at all, and either way the stored model adds a monthly charge per Custom Model Unit on top of whatever the serving meter reads.
Evaluation
Side by side
| Option | Live answer | Bill scales with tokens | Rate for the same tokens | Steady throughput | Idle hours are free | No commitment |
|---|---|---|---|---|---|---|
| On-demand invocation | ✓ | ✓ | baseline | ✗ | ✓ | ✓ |
| Batch inference | ✗ | ✓ | half | ✗ | ✓ | ✓ |
| Provisioned Throughput | ✓ | ✗ | flat per hour | ✓ | ✗ | optional |
| Fine-tune on a custom model deployment | ✓ | ✓ | per token | ✗ | ✓ | ✓ |
| Imported weights (Custom Model Import) | ✓ | ✗ | per active minute | ✓ | ✓ | ✓ |
Read the second and fifth columns together and the shape of the decision appears. Where the bill scales with tokens, an idle hour is free; where capacity is held by the hour, the quiet hours are charged as well. Imported weights sit between the two, metering time rather than tokens, but only the time a copy is running. Nothing here is cheaper in general; each option is cheaper for a particular pattern of use, and a workload made of several patterns is overpaying whenever it puts all of them through one option.
The batch row is the one teams skip past, because a column of two crosses looks weak. Those crosses are the absence of things the archive job never needed. Half the rate for identical tokens is the largest saving in this comparison.
Which way the decision runs
The solution
Split the workload by who is waiting, then fix the tokens.
The Summarise button and its follow-up questions stay on on-demand invocation. Three thousand presses a day is spiky work, clustered in office hours and dead overnight, and it needs an answer while somebody watches. Paying only for the tokens that are actually used, with nothing charged for the fourteen quiet hours, matches that shape better than any committed capacity would.
The nightly archive run and the sixty-thousand-document backlog move to batch inference. Neither has a person attached to it, both already read their input from and write their output to storage, and the same tokens cost half as much submitted as a batch job. This is the single largest saving available here, and it changes nothing a caseworker would notice.
Provisioned Throughput is not worth committing to yet. It starts to make sense when the flat hourly cost of a model unit comes in under the on-demand cost of the tokens flowing through it. This workload is nowhere near that line, and the sum below shows by how much. It is also the meter the team would land on if it later fine-tunes a model on the broker’s own claim-note style and picks a base outside the short list that supports an on-demand custom model deployment. Neither decision is due yet, and making a term commitment before the volume is real is how teams end up paying for idle capacity for six months.
Before any of that, three changes cut the bill without changing the option. Trim the system prompt, because every word of it is charged on every one of six thousand interactive calls a day. Set a maximum output length on the summary, which pulls cost and latency down together. And stop resending the full twelve pages on every follow-up: send the summary plus the passages the question actually needs, and the follow-up traffic stops costing as much as the summaries do. Budgeting tokens across a long-document workload takes that further than a foundational treatment needs to.
Two housekeeping items make the next bill explicable. Pick the Region deliberately, checking regional coverage for the models actually shortlisted rather than assuming the nearest one has them. If throttling shows up in the logs and the data is permitted in more than one Region, consider a cross-Region inference profile. Then put the workload behind AWS Budgets with an alert, and use AWS Cost Explorer to see which part of it is growing. A bill nobody can attribute is how a team ends up four times over an estimate without noticing for a month.
Worked example
Take illustrative rates of USD$0.80 per million input tokens and USD$4.00 per million output tokens. AWS publishes Bedrock rates in US dollars, and they differ by model, by Region, and over time, so the arithmetic matters more than these particular numbers.
One summary. Six thousand words of document is about 8,000 tokens, plus a 200-token system prompt, so 8,200 input tokens. A four-hundred-word summary is about 530 output tokens.
- Input: 8,200 × USD$0.80 ÷ 1,000,000 = USD$0.0066
- Output: 530 × USD$4.00 ÷ 1,000,000 = USD$0.0021
- Per summary: about USD$0.0087, so a shade under a cent.
One day of the button. Three thousand summaries at USD$0.0087 is USD$26. Now the follow-ups: a third of those documents get three questions each, which is three thousand extra calls. Each one resends the document, the summary, and the conversation so far, so call it 9,300 input tokens for a 150-token answer, which is USD$0.0074 input and USD$0.0006 output, about USD$0.0080 a call. Three thousand of those is another USD$24.
The follow-up traffic generates about a quarter as much text as the summaries and costs nearly as much, because it pays for the same twelve pages over and over. This is what turned a whiteboard estimate into a surprise.
The archive. Two thousand documents a night at USD$0.0087 is USD$17 a night, or roughly USD$520 a month, and about USD$260 on batch inference. The sixty-thousand-document backlog is USD$522 on-demand and about USD$261 as a batch job, paid once.
And the Provisioned Throughput sanity check. Thirty days of the button, the follow-ups, and a batched archive run to roughly USD$1,800 a month of tokens. If a model unit costs (again illustratively) USD$30 an hour, one unit running continuously is about USD$21,600 a month. The commitment would have to shrink the bill by a factor of twelve to break even, which no amount of negotiation achieves. Nothing about this workload is close to the volume where holding capacity beats paying per token.
What’s worth remembering
- Under the token-based pricing model you pay separately for input tokens and output tokens, and the output rate is usually several times the input rate.
- A model keeps nothing between calls, so every prompt is charged in full every time, and a long exchange about one document pays for that document again on every turn.
- Output length drives latency as well as cost, because the model generates one token at a time, so capping the answer improves both at once.
- Batch inference charges fifty per cent below the on-demand rate for identical tokens, in return for a job window measured in hours instead of a live answer.
- Provisioned Throughput swaps a per-token bill for a per-hour one that is cheaper only while the capacity stays busy, and it is the only route to a fine-tune whose base model has no on-demand deployment option.
- Regional coverage, availability, and redundancy are cost decisions too: the Regions your data may use narrow which models you can pick, and a cross-Region inference profile raises the ceiling under load at no extra routing charge.