The situation
An insurance broker runs a claims team of forty caseworkers. Every claim arrives with paperwork: medical reports, repair quotes, incident statements, letters from solicitors. A typical document runs twelve pages, around six thousand words, and a caseworker used to read all of it to find four facts.
In June the team put a Summarise button in the case-management screen. It sends the document to a foundation model on Amazon Bedrock and returns a four-hundred-word summary. Caseworkers press it about three thousand times a day, and roughly a third of the time they follow up with three more questions about the same document without leaving the screen. Separately, a nightly job summarises two thousand documents from the archive so older claims become searchable, and a one-off backlog of sixty thousand archived documents is queued for the same treatment.
The first full month’s Bedrock bill came to a little over two thousand dollars against an estimate of five hundred that somebody had done on a whiteboard. Nobody on the team can say which part of the workload produced it, because nobody can say what a single call costs. That is the first thing to fix. The second is deciding how each part of this workload should be paid for.
What actually matters
Start with the unit. A token is a piece of text slightly shorter than a word. Common words are one token each, longer or unusual ones split into two or three, and a thousand tokens comes out at roughly seven hundred and fifty words of ordinary English. Models read and generate in tokens, and Bedrock meters them. That is the token-based pricing model: you are charged for the input tokens you send and the output tokens the model generates, and nothing else about the call is metered. Not the number of requests, not the wall-clock time, not the size of the model behind it. Two rates apply rather than one, and the output rate is usually several times the input rate, because producing text costs the model far more work than reading it.
The second thing to understand is that every call is charged in full, every time. A foundation model holds nothing between calls, so anything it needs to see has to be sent again. The Summarise button sends the whole twelve-page document. A follow-up question sends that document again, plus the summary, plus the earlier questions and answers, plus the new question. By the third follow-up the team has paid for that document four times. Everything that occupies the context window is charged at the input rate on every call that carries it, which puts a standing price on a long system prompt and on the habit of replaying an entire conversation verbatim.
Output tokens do double duty, and this is where cost and performance meet. A model generates one token at a time, in order, so a nine-hundred-word answer takes about three times as long to arrive as a three-hundred-word one and costs about three times as much. Capping the length of the answer improves both numbers with one setting, which no other lever here does. Where a wait is unavoidable, streaming tokens back as they are generated changes how the wait feels without changing what it costs.
Then there is who is waiting. Responsiveness is a property of the work rather than of the model. A caseworker watching a spinner needs an answer in seconds; the archive job does not care whether a document finishes at 2am or 5am, and nobody will ever know which. That difference is worth money, because Bedrock charges less per token for work it is allowed to schedule around its own capacity.
The last set of concerns is where the capacity comes from. On-demand capacity is shared and governed by account quotas, so a burst past the limit is throttled. That is availability. Retries with backoff, and somewhere else to send the traffic, are the redundancy. Regional coverage is a harder constraint than teams expect. A given model is not offered in every AWS Region, so if this claims data may only sit in one country, the catalogue available to the workload is whatever that Region offers. A cross-Region inference profile spreads one workload’s calls across several Regions to raise the ceiling under load, which helps only if every Region in the profile is one the data is allowed to enter.
What we’ll filter on
AWS names the cost trade-offs of its GenAI services as responsiveness, availability, redundancy, performance, regional coverage, token-based pricing, provision throughput, and custom models. Applied to this workload, that list becomes six things to compare the options on.
- Responsiveness: is a person waiting on this call, and what breaks if the answer arrives in four hours instead of four seconds?
- What the bill scales with: the tokens consumed, or the time capacity is held whether or not anything flows through it?
- The rate on identical tokens: does this option charge more, less, or the same for the same input and output?
- Availability and redundancy: what happens in the busiest hour, and is there anywhere else for the traffic to go?
- Regional coverage: is the model offered in the Regions this data is permitted to sit in?
- Commitment: how long are we tied in, and what does an idle hour cost?
The landscape
There are four ways to pay for inference on Bedrock, the last of which splits in two depending on where the weights came from, and a workload with several shapes of job in it will usually want more than one of them.
On-demand invocation
The default, and what the Summarise button is doing today. Call the model, get an answer back in the same request, pay the published per-token rate for that model in that Region. No commitment, nothing to provision, and an hour with no traffic costs nothing at all. Throughput sits under account quotas shared with everything else calling that model in that Region, so a burst past the limit is throttled and the application has to retry. Latency varies with how busy the shared pool is.
Batch inference
Write the requests into a file in Amazon S3, submit one job, and collect the results from S3 when it finishes. Bedrock runs the job as capacity allows, typically inside a day, and charges roughly half the on-demand rate for exactly the same tokens. There is no live response and no per-request latency worth quoting, because the unit of work is a job with a completion window rather than a request with a timeout. The trade is the same one that serving a trained model on Amazon SageMaker AI presents in its own vocabulary: work nobody is waiting for should not be paying for capacity that stands ready.
Provisioned Throughput
Buy capacity rather than tokens. You purchase model units, each giving a guaranteed number of tokens per minute, either by the hour with no commitment or at a lower rate with a one-month or six-month term. The bill is then flat. It is the same number whether the units run hot all day or sit idle from midnight. The arithmetic works only when sustained volume keeps them busy for most of the hours being paid for. What comes with it is throughput and latency that do not move when the shared pool gets busy. The commitment maths, and the capacity tiers that have grown up alongside this option, are worked through at length in matching Bedrock pricing to a workload’s rhythm.
Custom models and their hosting
A custom model is either a base model you have changed or a set of weights you have brought with you, and the two are served on different meters. A Bedrock fine-tune is served as its base model allows: some customised bases stay callable on demand at a per-token rate, and others have no on-demand path at all, so the only way to reach them is Provisioned Throughput bought against the base model’s units and charged for every hour it is held. Weights trained outside Bedrock take a different route. Custom Model Import puts them behind the same API and meters them by the Custom Model Unit, billed for the five-minute windows in which a copy is active, scaling to zero when nothing calls it and paying a cold start of tens of seconds on the call that wakes it again. Provisioned Throughput is not offered for imported weights at any price. Training a fine-tune or importing weights is a one-off charge, and an imported model carries a monthly storage charge as well, both on top of whatever the serving meter reads.
Evaluation
Side by side
| Option | Live answer | Bill scales with tokens | Rate for the same tokens | Steady throughput | Idle hours are free | No commitment |
|---|---|---|---|---|---|---|
| On-demand invocation | ✓ | ✓ | baseline | ✗ | ✓ | ✓ |
| Batch inference | ✗ | ✓ | about half | ✗ | ✓ | ✓ |
| Provisioned Throughput | ✓ | ✗ | flat per hour | ✓ | ✗ | ✗ |
| Bedrock fine-tune with no on-demand path | ✓ | ✗ | flat per hour | ✓ | ✗ | ✗ |
| Imported weights (Custom Model Import) | ✓ | ✗ | per active minute | ✓ | ✓ | ✓ |
Read the second and fifth columns together and the shape of the decision appears. Where the bill scales with tokens, an idle hour is free; where it buys capacity by the hour, the quiet hours are charged as well. Imported weights sit between the two, metering time rather than tokens but only the time a copy is awake. Nothing here is cheaper in general; each option is cheaper for a particular pattern of use, and a workload made of several patterns is overpaying whenever it puts all of them through one option.
The batch row is the one teams skip past, because a column of two crosses looks weak. Those crosses are the absence of things the archive job never needed. Half price for identical tokens, in exchange for a property the work does not have, is the easiest money in this whole comparison.
Which way the decision runs
The solution
Split the workload by who is waiting, then fix the tokens.
The Summarise button and its follow-up questions stay on on-demand invocation. Three thousand presses a day is spiky work, clustered in office hours and dead overnight, and it needs an answer while somebody watches. Paying only for the tokens that are actually used, with nothing charged for the fourteen quiet hours, matches that shape better than any committed capacity would.
The nightly archive run and the sixty-thousand-document backlog move to batch inference. Neither has a person attached to it, both already read their input from and write their output to storage, and the same tokens cost about half as much when Bedrock is allowed to schedule the work. This is the single largest saving available here, and it changes nothing a caseworker would notice.
Provisioned Throughput stays on the shelf. It starts to make sense when the flat hourly cost of a model unit comes in under the tokens flowing through it. This workload is nowhere near that line, and the sum below shows by how much. It is also the meter the team would land on if it later fine-tunes a model on the broker’s own claim-note style and the base it picks offers no on-demand path for customised versions. Neither decision is due yet, and making a term commitment before the volume is real is how teams end up paying for idle capacity for six months.
Before any of that, three changes cut the bill without changing the option. Trim the system prompt, because every word of it is charged on every one of five thousand calls a day. Set a maximum output length on the summary, which pulls cost and latency down together. And stop resending the full twelve pages on every follow-up: send the summary plus the passages the question actually needs, and the follow-up traffic stops costing as much as the summaries do. Budgeting tokens across a long-document workload takes that further than a foundational treatment needs to.
Two housekeeping items make the next bill explicable. Pick the Region deliberately, checking regional coverage for the models actually shortlisted rather than assuming the nearest one has them. If throttling shows up in the logs and the data is permitted in more than one Region, consider a cross-Region inference profile. Then put the workload behind AWS Budgets with an alert, and use AWS Cost Explorer to see which part of it is growing. A bill nobody can attribute is how a team ends up four times over an estimate without noticing for a month.
Worked example
Take illustrative rates of $0.80 per million input tokens and $4.00 per million output tokens. Real rates differ by model, by Region, and over time, so the arithmetic matters more than these particular numbers.
One summary. Six thousand words of document is about 8,000 tokens, plus a 200-token system prompt, so 8,200 input tokens. A four-hundred-word summary is about 530 output tokens.
- Input: 8,200 × $0.80 ÷ 1,000,000 = $0.0066
- Output: 530 × $4.00 ÷ 1,000,000 = $0.0021
- Per summary: about $0.0087, so a shade under a cent.
One day of the button. Three thousand summaries at $0.0087 is $26. Now the follow-ups: a third of those documents get three questions each, which is three thousand extra calls. Each one resends the document, the summary, and the conversation so far, so call it 9,300 input tokens for a 150-token answer, which is $0.0074 input and $0.0006 output, about $0.0080 a call. Three thousand of those is another $24.
That second number is the one worth staring at. The follow-up traffic produces about a twentieth as much text as the summaries and costs nearly as much, because it pays for the same twelve pages over and over. This is what turned a whiteboard estimate into a surprise.
The archive. Two thousand documents a night at $0.0087 is $17 a night, or roughly $520 a month, and about $260 on batch inference. The sixty-thousand-document backlog is $522 on-demand and about $261 as a batch job, paid once.
And the Provisioned Throughput sanity check. The whole workload above runs to roughly $1,500 a month of tokens. If a model unit costs (again illustratively) $30 an hour, one unit running continuously is about $21,600 a month. The commitment would have to shrink the bill by a factor of fourteen to break even, which no amount of negotiation achieves. Nothing about this workload is close to the volume where holding capacity beats paying per token.
What’s worth remembering
- Under the token-based pricing model you pay separately for input tokens and output tokens, and the output rate is usually several times the input rate.
- A model keeps nothing between calls, so every prompt is charged in full every time, and a long exchange about one document pays for that document again on every turn.
- Output length drives latency as well as cost, because the model generates one token at a time, so capping the answer improves both at once.
- Batch inference charges about half the on-demand rate for identical tokens, and asks in return only that nobody is waiting.
- Provisioned Throughput, and a fine-tune that can only be served on it, replace a per-token bill with a per-hour one that is cheaper only when the capacity is busy for most of the hours you pay for, while imported weights meter the minutes a copy is active and scale to zero in between.
- Regional coverage, availability, and redundancy are cost decisions too: the Regions your data may use decide which models you can pick, and a cross-Region inference profile trades a little latency for headroom under load.