The situation
A building-services contractor sends about 2,400 engineers out to 31,000 maintenance visits a week. Every visit produces a job report: what was found, what was done, what still needs doing, typed into a phone in the van and running to around 1,100 words. Account managers read these before their monthly client calls, which means somebody reads several hundred of them in a bad week.
The proposal is a Summarise button. It sends a job report to a foundation model on Amazon Bedrock and returns a 120-word handover paragraph. Around 4,400 a day would be pressed by coordinators who are waiting for the answer on screen. The rest, roughly 9,000 a week, would run overnight so the account manager finds them ready in the morning. There is also a backlog of 900,000 archived reports that somebody wants summarised once, so the archive becomes browsable.
The complication is on the procurement form. The company published a target two years ago: a 46 per cent cut in absolute emissions by 2032 against a 2022 baseline, audited annually. Cloud spending lands in scope 3, the emissions the company causes but does not directly produce. Since the target was published, every new system above a spending threshold has to state its expected energy and emissions contribution before sign-off. The sponsor has to fill that box in. Nobody on the team has ever had to answer that question about a model, and the two answers on the table so far are “the biggest model, because it is best” and “we cannot say”.
What actually matters
Two objectives sit next to each other in AWS’s guidance for this domain, and reading them together explains why the procurement box exists. One asks you to identify features of responsible AI, which the material lists as “bias, fairness, inclusivity, robustness, safety, veracity”: the things a model is judged on once it is running. The other asks for responsible practices to select a model, and gives environmental considerations and sustainability as its examples. That second one is about the choosing rather than the output, and it is the one this scenario turns on.
The mechanism underneath it is simpler than it sounds. Generating a token means reading a large number of model parameters out of memory and doing arithmetic with them, and that work is what draws power. Two things scale it. The first is model size: a model with a tenth as many active parameters does roughly a tenth as much arithmetic for each token it produces. The second is how many tokens pass through, counting both what you send and what comes back. Those two multiply. A frontier model summarising 1,100 words of job report uses far more energy than a small model summarising the same report. Both use more if the prompt carries four pages of instructions that nobody has trimmed since the prototype. Everything else in this decision is a variation on those two numbers.
Training is the third number, and it dwarfs the other two in one particular situation. Pre-training a foundation model from scratch means thousands of accelerators running continuously for weeks or months. Adapting an existing one, whether by fine-tuning it on your own examples or simply by writing a better prompt, is a handful of machines for hours, or no machines at all. The gap between those is not a percentage, it is several orders of magnitude. Reusing a model somebody has already trained is the largest single reduction available here, and the team already has it the moment it calls Bedrock. Name it on the form anyway. A sponsor who does not know that reduction has already been taken can be talked into a bespoke model later, on the grounds that it would be more accurate.
Then idle capacity, which is the one that surprises people. Amazon Bedrock sells Provisioned Throughput: dedicated capacity bought in model units, each rated for a number of input and output tokens a minute. It is billed hourly, with no commitment or on a one-month or six-month term, and the longer term discounts the hourly price. Hardware reserved for you is powered and cooled whether or not requests arrive. A workload that runs hot from 7am to 6pm and does nothing overnight pays for, and consumes, a full day of it. On-demand invocation shares a pool across many customers, so the gaps in this workload are filled by somebody else’s traffic and the underlying machines run closer to full. Higher utilisation of the same hardware is the same lever a data centre operator pulls, and choosing on-demand is how a small workload pulls it. Matching a pricing shape to a workload rhythm is usually argued on money; the energy argument runs in the same direction, which makes it an easy one to win.
Finally, the split of who controls what. The sustainability pillar of the AWS Well-Architected Framework states it as sustainability of the cloud against sustainability in the cloud. AWS owns the first: shared, efficient infrastructure, cooling, water stewardship, sourcing renewable power, hardware refresh. You own the second, which AWS words as minimising the total resources your workloads need. How large a model you invoke, how many tokens you send, how much capacity you hold idle, which AWS Region you run in, and how long you keep data you will never read again. Nothing on the customer side of that line is reduced by a vendor announcement, and everything on it is reduced by decisions taken at design time.
One caution before the options. Environmental impact is one filter among several, and a model that halves the energy and doubles the error rate has not made the system more responsible. Summaries that are wrong get an engineer sent back to a site in a van, which has its own emissions and its own damage to the client relationship. The sustainability argument only holds when the small model actually does the job.
What we’ll filter on
- Task fitness. Does it produce a handover paragraph an account manager can use without checking the source report?
- Cost per request. What does one summary cost at the volumes above, live and in bulk?
- Latency. Can it answer a coordinator who is waiting on screen, and does response time matter at all for the overnight run?
- Energy and carbon footprint. How much compute does one request draw, and does anything draw power while no requests arrive?
- Reversibility. If this turns out to be wrong in three months, how much work is it to change course?
The landscape
The options run from a service that does one narrow thing very cheaply to a model we would train ourselves, and the energy profile changes by orders of magnitude across that range.
A large frontier model on Amazon Bedrock
The default proposal, and the strongest summariser in the catalogue. It handles a badly typed job report with abbreviations, part numbers and half sentences, and produces something readable on the first attempt with a short prompt. Bedrock charges per token, so nothing is reserved and an idle hour costs nothing.
What comes with it is the largest per-token energy draw of anything here, because the most parameters have to be read for every token generated, and the highest price per token to match. For a task where a smaller model would also succeed, that is compute used for capability the task never draws on. Frontier models also carry the widest capability surface, which matters when weighing what a model is actually needed for: this one needs summarisation of short technical prose in one language, and nothing else.
A smaller model in the same family
Model families ship in sizes. The Amazon Nova understanding models, for example, run from Micro, which is text-only and the lowest latency of the set, through Lite and Pro to Premier, all reachable through the same Bedrock API with the same request shape. Dropping down a size changes one line of configuration.
Smaller models are quicker, cheaper per token and lighter on energy, all for the same reason: fewer parameters to read per token. They are also worse at hard prompts, and how much worse depends entirely on the task. Summarising a structured job report into a fixed-shape paragraph sits near the easy end of what a language model does. On tasks like that, the gap between sizes tends to collapse. Whether it collapses far enough here is a measurement, not an opinion, and running the same evaluation set against both sizes is how it gets settled.
A distilled model
Distillation trains a small model to imitate a larger one on a specific kind of work. A teacher model answers a few thousand real job reports, and the student is fine-tuned on those pairs. Amazon Bedrock Model Distillation runs the whole sequence as one managed job, from prompts you supply or from your existing CloudWatch invocation logs, so production traffic can be the training data.
The result can be close to the large model’s quality at the small model’s per-token energy. Two things come with it. The distillation run is itself compute, and it has to be repeated whenever the task or the source model changes. The distilled model is then a custom model, and Bedrock serves those two ways: Provisioned Throughput, held and billed by the hour, or a custom model deployment for on-demand inference. The second is limited to US East (N. Virginia) and US West (Oregon) and to a short list of base models, so a workload pinned to Europe takes the provisioned route. Capacity held through a quiet overnight period can draw more energy in total than a stock small model called on demand, even where each individual request is lighter. This is worth reaching for when a stock small model has been measured and genuinely falls short.
An open-weight model on Amazon EC2 or Amazon SageMaker AI
Take a published open-weight model, from Amazon SageMaker JumpStart or elsewhere, and run it on accelerated instances you control. Full control over the model version, the hardware and where it runs, no per-token pricing, and the option to run it in a Region or an account with particular constraints.
The instances run whether traffic arrives or not, which makes this workload’s energy profile a function of utilisation. At 4,400 live requests a day spread over a working day, a single accelerated instance sits mostly idle, and idle accelerated hardware draws a substantial share of its busy power. Achieving good utilisation means batching aggressively, scaling to zero out of hours, or having far more traffic than this. It also transfers a pile of work to the team: patching, model updates, capacity planning, autoscaling. This one is a fit when volume is high and steady, or when a constraint rules the managed services out.
A purpose-built AWS AI service
Amazon Comprehend does natural-language processing against pre-trained models, with no model choice to make: entities, key phrases, sentiment, targeted sentiment, PII, dominant language and syntax, over an API, at a fraction of the cost and compute of a foundation model. Amazon Textract does the equivalent for reading text, forms and tables out of documents.
Comprehend does not write a paragraph. It extracts, classifies and labels. Suppose the account managers turned out to need three facts rather than a written handover: which assets were touched, what parts were used, whether anything is outstanding. Comprehend covers that, at the smallest footprint on this list by a wide margin. The lesson generalises past this one scenario: a lot of work handed to a large language model is extraction with a prose wrapper around it, and checking whether the task needs generated text at all belongs before any model comparison starts.
Pre-training our own model
Listed so it can be dismissed with a number rather than a shrug. Training a model from scratch on maintenance-industry text would take thousands of accelerator-months and cost more than the feature will ever return. It would also be worse than a stock model, which was trained on far more English than this company will ever write. Customising an existing model covers everything this company might plausibly need, for a tiny fraction of the energy. Pre-training makes sense for organisations building foundation models as a product, and for essentially nobody else.
Evaluation
Side by side
| Option | Clears the evaluation bar | Cost per request | Latency | Energy per request | Draws power when idle | Reversible |
|---|---|---|---|---|---|---|
| Large frontier model on Bedrock | ✓ | highest | slowest | highest | ✗ | ✓ |
| Smaller model in the same family | to be measured | low | fast | low | ✗ | ✓ |
| Distilled model | ✓ (after a training run) | medium | fast | low per call | ✓ | partly |
| Open-weight model on EC2 or SageMaker AI | ✓ | volume-dependent | yours to tune | low per call | ✓ | ✗ |
| Amazon Comprehend | ✗ for prose summaries | lowest | fastest | lowest | ✗ | ✓ |
| Pre-train our own | unlikely | enormous | n/a | enormous | ✓ | ✗ |
Two columns do most of the work. The energy column falls away sharply as model size drops, and the idle column separates the options that consume only when used from the options that consume by the hour. An option that is cheap per call and always warm can lose to an option that is dearer per call and cold, and at this workload’s shape it does.
Which gate the decision trips
The measurement that settles gate two took a day. Two hundred job reports went through the large model and through the smallest model in the same family. Three account managers scored the pairs blind, on whether they would have made the client call from the summary alone. The large model scored 94 per cent usable, the small model 89. Both failures were the same shape: reports where the engineer had typed almost nothing, and no model can summarise an empty page. On the reports that had content in them, the two sizes were within a point of each other.
The solution
Run the smallest model in the family that cleared the bar, on demand, with the token count trimmed and the overnight work sent through batch inference.
Start with the model size, because it moves the energy figure furthest. Five points of usability, 94 against 89, is worth arguing about. The argument resolves once both numbers are read against the threshold the account managers set, which was that the summary saves a read of the full report. Write both numbers on the sign-off form with the energy comparison next to them, so a future reviewer sees the trade that was made rather than a preference that was asserted.
Then the tokens, since energy scales with them as directly as it scales with model size. The prototype prompt carried 900 words of instructions and three worked examples on every single call. Cutting it to 200 words with one example removed about a third of the input tokens outright, with no measurable change in output quality. Prompt caching handles what is left: the instruction block is identical on every call, so caching it means Bedrock reuses the processed prefix instead of recomputing it, which cuts both the bill and the work done. That applies to the live path only, because Bedrock does not support prompt caching on the batch inference API. Capping output at 200 tokens, a shade above the 160 or so a 120-word paragraph needs, stops the model writing 400 words when 120 were asked for.
The overnight 9,000 and the 900,000-report backlog go through batch inference. Nobody is waiting, so there is no reason to hold live capacity for them, and Bedrock schedules the job into whatever capacity is free. AWS prices batch at 50 per cent below the on-demand rate on the models that support it, and it lets the provider run its fleet at a higher average utilisation, which is the same efficiency argument the idle column made in reverse. The backlog will not fit one job: an input file tops out at 1 GB and a job at 5 GB across its files, so 900,000 reports become a run of jobs. Live coordinator traffic stays on on-demand invocation. Neither path uses Provisioned Throughput, because 4,400 requests a day spread across a working day would not keep a reserved unit busy and would bill for the thirteen hours it sat cold. Revisit that if the volume grows by an order of magnitude, and revisit it with utilisation numbers rather than a hunch.
Region choice is the one lever most teams never touch. AWS Regions draw on different electricity grids, and the carbon intensity of those grids varies by a large multiple. The sustainability pillar puts it as a two-step: shortlist Regions on compliance, available features, cost and latency, then choose from that shortlist the one nearest Amazon’s renewable energy projects or on a grid with a lower published carbon intensity. Here the job reports are UK data with no residency rule beyond keeping them in the UK or Europe, and the model is offered in more than one European Region, so the shortlist has more than one name on it. Where data residency does pin the Region, that constraint wins and the honest thing is to record it rather than pretend the choice existed.
No pre-training, and no fine-tuning yet. The stock model with a good prompt already clears the bar, so a customisation run would burn compute on a problem the measurement says is not there. Keep it in reserve for the day the evaluation shows a gap that prompting cannot close.
Last, make the figure reportable. Create an application inference profile for the Summarise feature and tag it, because those tags are what carry Bedrock on-demand usage into cost allocation, and at a fixed per-token rate the spending is a faithful proxy for tokens processed. The auditors will want reported emissions rather than a proxy, and those now come from the AWS Sustainability console, which replaced the Customer Carbon Footprint Tool when AWS deprecated it on 30 June 2026. It breaks carbon down by scope, Region, service and account, monthly, published by the 21st of the month after the usage. Run the sustainability pillar review in the AWS Well-Architected Tool once the feature is live, and put the date of the next one in the diary. A feature nobody measures after launch drifts in every dimension, and this one is now attached to a published target.
What’s worth remembering
- Inference energy scales with model size and with tokens processed, so the smallest model that clears the evaluation bar, on the shortest prompt that works, cuts more than any other choice available to you.
- Pre-training a model from scratch uses orders of magnitude more energy than customising an existing one, which makes reusing a pre-trained model the first of the responsible practices to select a model.
- Provisioned or self-hosted capacity draws power whether requests arrive or not, so on-demand and batch inference suit a workload with quiet hours better than reserved capacity does.
- Batch inference for work nobody is waiting on cuts both the bill and the compute, because the provider can schedule it into capacity that would otherwise sit idle.
- Under the shared responsibility model, AWS owns data-centre and hardware efficiency while you own how much compute you request, which is what the sustainability pillar of the AWS Well-Architected Framework asks you to minimise.
- Environmental considerations sit alongside accuracy, cost and latency rather than above them, because a cheaper model that gets the answer wrong sends a van back to site.