The situation
A building-services contractor sends about 2,400 engineers out to 31,000 maintenance visits a week. Every visit produces a job report: what was found, what was done, what still needs doing, typed into a phone in the van and running to around 1,100 words. Account managers read these before their monthly client calls, which means somebody reads several hundred of them in a bad week.
The proposal is a Summarise button. It sends a job report to a foundation model on Amazon Bedrock and returns a 120-word handover paragraph. Around 4,400 a day would be pressed by coordinators who are waiting for the answer on screen. The rest, roughly 26,000 a week, would run overnight so the account manager finds them ready in the morning. There is also a backlog of 900,000 archived reports that somebody wants summarised once, so the archive becomes browsable.
The complication is on the procurement form. The company published a target two years ago: a 46 per cent cut in absolute emissions by 2032 against a 2022 baseline, audited annually. Cloud spending lands in scope 3, the emissions the company causes but does not directly produce. Since the target was published, every new system above a spending threshold has to state its expected energy and emissions contribution before sign-off. The sponsor has to fill that box in. Nobody on the team has ever had to answer that question about a model, and the two answers on the table so far are “the biggest model, because it is best” and “we cannot say”.
What actually matters
Two objectives sit next to each other in AWS’s guidance for this domain, and reading them together explains why the procurement box exists. One asks you to identify features of responsible AI, which the material lists as “bias, fairness, inclusivity, robustness, safety, veracity”: the things a model is judged on once it is running. The other asks for responsible practices to select a model, and gives environmental considerations and sustainability as its examples. That second one is about the choosing rather than the output, and it is the one this scenario turns on.
The mechanism underneath it is simpler than it sounds. Generating a token means reading a large number of model parameters out of memory and doing arithmetic with them, and that work is what draws power. Two things scale it. The first is model size: a model with a tenth as many active parameters does roughly a tenth as much arithmetic for each token it produces. The second is how many tokens pass through, counting both what you send and what comes back. Those two multiply. A frontier model summarising 1,100 words of job report costs far more energy than a small model summarising the same report. Both cost more if the prompt carries four pages of instructions that nobody has trimmed since the prototype. Everything else in this decision is a variation on those two numbers.
Training is the third number, and it dwarfs the other two in one particular situation. Pre-training a foundation model from scratch means thousands of accelerators running continuously for weeks or months. Adapting an existing one, whether by fine-tuning it on your own examples or simply by writing a better prompt, is a handful of machines for hours, or no machines at all. The gap between those is not a percentage, it is several orders of magnitude. Reusing a model somebody has already trained is the largest single reduction available here, and the team gets it for free by using Bedrock at all. Name it on the form anyway. A sponsor who does not know that reduction has already been taken can be talked into a bespoke model later, on the grounds that it would be more accurate.
Then idle capacity, which is the one that surprises people. Amazon Bedrock will sell you Provisioned Throughput: dedicated capacity, reserved by the hour or on a longer term, giving guaranteed tokens per minute. Hardware reserved for you is powered and cooled whether or not requests arrive. A workload that runs hot from 7am to 6pm and does nothing overnight pays for, and consumes, a full day of it. On-demand invocation shares a pool across many customers, so the gaps in this workload are filled by somebody else’s traffic and the underlying machines run closer to full. Higher utilisation of the same hardware is the same lever a data centre operator pulls, and choosing on-demand is how a small workload pulls it. Matching a pricing shape to a workload rhythm is usually argued on money; the energy argument runs in the same direction, which makes it an easy one to win.
Finally, the split of who controls what. The AWS shared responsibility model has a sustainability version, set out in the sustainability pillar of the AWS Well-Architected Framework. AWS owns the efficiency of everything below the service boundary: data-centre design, cooling, power purchasing, hardware refresh, the efficiency of its own silicon. You own the demand you place on it. How large a model you invoke, how many tokens you send, how much capacity you hold idle, which AWS Region you run in, and how long you keep data you will never read again. Nothing on the customer side of that line is reduced by a vendor announcement, and everything on it is reduced by decisions taken at design time.
One caution before the options. Environmental cost is one filter among several, and a model that halves the energy and doubles the error rate has not made the system more responsible. Summaries that are wrong get an engineer sent back to a site in a van, which has its own emissions and its own client-facing cost. The sustainability argument only holds when the small model actually does the job.
What we’ll filter on
- Task fitness. Does it produce a handover paragraph an account manager can use without checking the source report?
- Cost per request. What does one summary cost at the volumes above, live and in bulk?
- Latency. Can it answer a coordinator who is waiting on screen, and does the overnight work care?
- Energy and carbon footprint. How much compute does one request draw, and does anything draw power while no requests arrive?
- Reversibility. If this turns out to be wrong in three months, what does it cost to change our mind?
The landscape
The options run from a service that does one narrow thing very cheaply to a model we would train ourselves, and the energy profile changes by orders of magnitude across that range.
A large frontier model on Amazon Bedrock
The default proposal, and the strongest summariser in the catalogue. It handles a badly typed job report with abbreviations, part numbers and half sentences, and produces something readable on the first attempt with a short prompt. Bedrock charges per token, so nothing is reserved and an idle hour costs nothing.
What comes with it is the largest per-token energy draw of anything here, because the most parameters have to be read for every token generated, and the highest price per token to match. For a task where a smaller model would also succeed, that is compute spent on capability the task never uses. Frontier models also carry the widest capability surface, which matters when weighing what a model is actually needed for: this one needs summarisation of short technical prose in one language, and nothing else.
A smaller model in the same family
Model families ship in sizes. The Amazon Nova family, for example, runs from a very small text model up to the largest, all reachable through the same Bedrock API with the same request shape. Dropping down a size changes one line of configuration.
Smaller models are quicker, cheaper per token and lighter on energy, all for the same reason: fewer parameters to read per token. They are also worse at hard prompts, and how much worse depends entirely on the task. Summarising a structured job report into a fixed-shape paragraph sits near the easy end of what a language model does. On tasks like that, the gap between sizes tends to collapse. Whether it collapses far enough here is a measurement, not an opinion, and running the same evaluation set against both sizes is how it gets settled.
A distilled model
Distillation trains a small model to imitate a larger one on a specific kind of work. You send the large model a few thousand real job reports, keep its outputs, and train the small model on those pairs. Amazon Bedrock Model Distillation runs this as a managed job and produces a small model that behaves like the large one on the narrow slice it was taught.
The result can be close to the large model’s quality at the small model’s per-token energy. Three costs come with it. The distillation run is itself compute, and it has to be repeated whenever the task or the source model changes. The distilled model is then served as a custom model, which on Bedrock means provisioning throughput and paying for the hours it stays deployed rather than the tokens it processes. A distilled model kept warm for a workload with a quiet overnight period can draw more energy in total than a stock small model called on demand, even though every individual request is cheaper. This is worth reaching for when a stock small model has been measured and genuinely falls short.
An open-weight model on Amazon EC2 or Amazon SageMaker AI
Take a published open-weight model, from Amazon SageMaker JumpStart or elsewhere, and run it on accelerated instances you control. Full control over the model version, the hardware and where it runs, no per-token pricing, and the option to run it in a Region or an account with particular constraints.
The instances run whether traffic arrives or not, which puts this workload’s energy profile at the mercy of utilisation. At 4,400 live requests a day spread over a working day, a single accelerated instance sits mostly idle, and idle accelerated hardware draws a substantial share of its busy power. Achieving good utilisation means batching aggressively, scaling to zero out of hours, or having far more traffic than this. It also transfers a pile of work to the team: patching, model updates, capacity planning, autoscaling. This one is a fit when volume is high and steady, or when a constraint rules the managed services out.
A purpose-built AWS AI service
Amazon Comprehend does natural-language processing without any model choice at all: entities, key phrases, sentiment, language detection, over an API, at a fraction of the cost and compute of a foundation model. Amazon Textract does the same for reading structured content out of documents.
Comprehend does not write a paragraph. It extracts, classifies and labels. Suppose the account managers turned out to need three facts rather than a written handover: which assets were touched, what parts were used, whether anything is outstanding. Comprehend covers that, at the smallest footprint on this list by a wide margin. The lesson generalises past this one scenario: a lot of work handed to a large language model is extraction with a prose wrapper around it, and checking whether the task needs generated text at all belongs before any model comparison starts.
Pre-training our own model
Listed so it can be dismissed with a number rather than a shrug. Training a model from scratch on maintenance-industry text would take thousands of accelerator-months and cost more than the feature will ever return. It would also be worse than a stock model, which has already read far more English than this company will ever write. Customising an existing model covers everything this company might plausibly need, at a rounding error of the energy. Pre-training makes sense for organisations building foundation models as a product, and for essentially nobody else.
Evaluation
Side by side
| Option | Clears the evaluation bar | Cost per request | Latency | Energy per request | Draws power when idle | Reversible |
|---|---|---|---|---|---|---|
| Large frontier model on Bedrock | ✓ | highest | ~4s | highest | ✗ | ✓ |
| Smaller model in the same family | to be measured | low | ~1s | low | ✗ | ✓ |
| Distilled model | ✓ (after a training run) | medium | ~1s | low per call | ✓ | partly |
| Open-weight model on EC2 or SageMaker AI | ✓ | volume-dependent | ~2s | low per call | ✓ | ✗ |
| Amazon Comprehend | ✗ for prose summaries | lowest | <1s | lowest | ✗ | ✓ |
| Pre-train our own | unlikely | enormous | n/a | enormous | ✓ | ✗ |
Two columns do most of the work. The energy column falls away sharply as model size drops, and the idle column separates the options that consume only when used from the options that consume by the hour. An option that is cheap per call and always warm can lose to an option that is dearer per call and cold, and at this workload’s shape it does.
Which gate the decision trips
The measurement that settles gate two took a day. Two hundred job reports went through the large model and through the smallest model in the same family. Three account managers scored the pairs blind, on whether they would have made the client call from the summary alone. The large model scored 94 per cent usable, the small model 89. Both failures were the same shape: reports where the engineer had typed almost nothing, and no model can summarise an empty page. On the reports that had content in them, the two sizes were within a point of each other.
The solution
Run the smallest model in the family that cleared the bar, on demand, with the token count trimmed and the overnight work sent through batch inference.
Start with the model size, because it moves the energy figure furthest. Five points of usability, 94 against 89, is worth arguing about. The argument resolves once both numbers are read against the threshold the account managers set, which was that the summary saves a read of the full report. Write both numbers on the sign-off form with the energy comparison next to them, so a future reviewer sees the trade that was made rather than a preference that was asserted.
Then the tokens, since energy scales with them as directly as it scales with model size. The prototype prompt carried 900 words of instructions and three worked examples on every single call. Cutting it to 200 words with one example removed about a third of the input tokens outright, with no measurable change in output quality. Prompt caching handles what is left: the instruction block is identical on every call, so caching it means Bedrock stores the processed form and skips recomputing it, which cuts both the bill and the work done. Capping the output at 150 tokens stops the model writing 400 words when 120 were asked for.
The overnight 26,000 and the 900,000-report backlog go through batch inference. Nobody is waiting, so there is no reason to hold live capacity for them, and Bedrock schedules the job into whatever capacity is free. It charges around half the on-demand rate, and it lets the provider run its fleet at a higher average utilisation, which is the same efficiency argument the idle column made in reverse. Live coordinator traffic stays on on-demand invocation. Neither path uses Provisioned Throughput, because 4,400 requests a day spread across a working day would not keep a reserved unit busy and would pay for the fourteen hours it sat cold. Revisit that if the volume grows by an order of magnitude, and revisit it with utilisation numbers rather than a hunch.
Region choice is the one lever most teams never touch. AWS Regions draw on different electricity grids, and the carbon intensity of those grids varies by a large multiple. The sustainability pillar’s guidance is to place workloads in Regions with a lower-carbon grid where nothing else forbids it. Here the job reports are UK data with no residency rule beyond keeping them in the UK or Europe. The model is offered in more than one European Region, so the choice is genuinely open. Look at the published Region data before making it out of habit. Where data residency does pin the Region, that constraint wins and the honest thing is to record it rather than pretend the choice existed.
No pre-training, and no fine-tuning yet. The stock model with a good prompt already clears the bar, so a customisation run would spend compute to solve a problem the measurement says is not there. Keep it in reserve for the day the evaluation shows a gap that prompting cannot close.
Last, make the figure reportable. AWS Cost Explorer gives Bedrock spending broken down by model and, with cost allocation tags, by feature, and at a fixed per-token rate that spending is a faithful proxy for tokens processed. Tag the feature so that number can be pulled without a spreadsheet exercise. The AWS Customer Carbon Footprint Tool in the billing console gives the account’s reported emissions, which is what the auditors will actually want. Run the sustainability pillar review in the AWS Well-Architected Tool once the feature is live, and put the date of the next one in the diary. A feature nobody measures after launch drifts in every dimension, and this one now has a published target watching it.
What’s worth remembering
- Inference energy scales with model size and with tokens processed, so the smallest model that clears the evaluation bar, on the shortest prompt that works, cuts more than any other choice available to you.
- Pre-training a model from scratch costs orders of magnitude more energy than customising an existing one, which makes reusing a pre-trained model the first of the responsible practices to select a model.
- Provisioned or self-hosted capacity draws power whether requests arrive or not, so on-demand and batch inference suit a workload with quiet hours better than reserved capacity does.
- Batch inference for work nobody is waiting on cuts both the bill and the compute, because the provider can schedule it into capacity that would otherwise sit idle.
- Under the shared responsibility model, AWS owns data-centre and hardware efficiency while you own how much compute you request, which is what the sustainability pillar of the AWS Well-Architected Framework asks you to minimise.
- Environmental considerations sit alongside accuracy, cost and latency rather than above them, because a cheaper model that gets the answer wrong sends a van back to site.