The situation
A product team is standing up four features on Amazon Bedrock. The first is a high-volume ticket classifier that tags each incoming support message with one of eight labels. It runs millions of times a month, and every millisecond and fraction of a cent shows up in the bill. The second is a retrieval assistant that answers questions over the company handbook, which means turning documents and queries into vectors so the closest passages can be found. The third is a hard-reasoning helper that untangles multi-step policy questions where a wrong answer is expensive. The fourth is a marketing tool that generates product imagery from a text brief.
Right now all four route to the same flagship chat model, because that was the one someone enabled first and it clearly works. The classifier is paying flagship prices to pick between eight labels. The retrieval feature is asking a chat model to “find similar text” when it should be producing embeddings. The image feature does not work at all, because a text model has no image output modality. The bill is large, and the latency is worse than it needs to be on the two features that run most often.
Benchmarking every model in the catalogue by hand is not on the table, and the question underneath all four features is the same. Given the shape of this workload, which class of model does it need, and what is the smallest one that clears the bar?
What actually matters
The Bedrock catalogue is a spread of model classes built for different jobs, not a single quality ladder with the flagship on top. The first decision is which kind of model. A flagship chat model pointed at an embeddings job is the wrong tool, not an overspend on the right one.
Modality decides the most: what goes in and what comes out. A text-in, text-out task, an image-generation task, a task that reads images or video alongside text, and a task that produces vectors are four different capabilities. The Nova family splits along that line. Nova Micro is text-only with a 128K context window, while Nova Lite and Nova Pro take text, images and video in and return text, both with 300K windows. The family’s generation members, Nova Canvas for images and Nova Reel for video, are on the catalogue’s legacy list with an end-of-life date of 30 September 2026. Generation has passed to third parties: Stability AI for images, Luma Ray 2 for video. Anthropic’s Claude, Meta’s Llama and Mistral cover text, several of them with image input, and Amazon Titan and Cohere both offer dedicated embeddings models. Match modality first, because nothing else matters if the model cannot produce the shape of output the task needs.
Once modality is settled, the reasoning difficulty of the task is what justifies model size. A classifier picking one of eight labels, a sentiment call, a short extraction: these are easy judgements, and a small fast model such as Nova Micro clears them at a fraction of the cost and latency of a flagship. A judgement that easy raises a prior question, though. A fixed label set with labelled history behind it is classifier territory, and even the smallest foundation model costs more than not calling one. What keeps a task like this on a foundation model is the absence of that history, a label set that changes faster than a retraining cycle, or messages that need reading rather than pattern-matching. Multi-step policy deduction, tricky synthesis, a task where a subtle mistake is costly: these justify Nova Pro or a large Claude, because the extra capability changes the answer. Flagship tokens spent on the classifier change nothing, and small-model tokens spent on the hard reasoner produce a wrong answer.
Cost and latency move together and both track model size. Smaller models are cheaper per token and answer faster; larger ones cost more on both the input and the output side, and take longer. Pricing is per token, split between input and output, so a task with long inputs or verbose outputs costs far more on a large model than a short classification does. High call volume multiplies every one of those fractions. That is why the classifier’s model choice moves the bill more than the rarely-used hard reasoner’s does.
Context-window size is its own axis, and the models differ widely. A short classification needs almost none. A long-document summariser, or a retrieval feature stuffing passages into the prompt, needs a window that holds the largest realistic input. Picking a 300K-window model for a task that never exceeds a page adds cost without adding capability, and picking an 8K one for a task that routinely overflows it fails outright.
Two more axes decide the edges. Customisation matters when the task needs a house style or a domain vocabulary that prompting alone cannot pin down. Fine-tuning covers Nova Micro, Lite and Pro, several Llama sizes, and a short list of others, so that requirement narrows the field early. Region availability and model access form an operational gate: not every model is offered in every region, and an available model still has to be enabled for the account before any call to it succeeds. Lifecycle status is the third gate in that group. Once a model enters the Legacy state, new accounts cannot adopt it, existing accounts can lose access after fifteen days without a call, and no new Provisioned Throughput can be created for it. Generation is the clearest case: Amazon moved image and video twice, from Titan Image Generator to Nova Canvas and Nova Reel, then out to third parties.
What a published score settles
Most models arrive with published benchmarks attached, and it helps to know what each suite measures. MMLU and its successors probe broad knowledge across dozens of academic subjects, GSM8K multi-step arithmetic, and HumanEval code that has to run against a hidden test suite rather than merely look plausible. TruthfulQA measures resistance to confident falsehood on questions where the obvious answer is wrong. BOLD and BBQ sit on the bias side, sampling how outputs shift when the subject’s demographic changes. Above these sit the leaderboards and arenas that aggregate several suites, often with human preference votes, into one ranking.
A score from any of them narrows a shortlist and never closes it. The sets are public, so they may sit in a model’s training data, and a model that leads on multi-subject knowledge can still mislabel the eight categories a particular ticket queue uses. No benchmark covers latency, per-token price, region availability, or context window. Treat the published numbers as a quick first pass that takes eight candidates down to three, then run an Amazon Bedrock evaluation job over a golden set drawn from the workload’s own traffic. Alignment with a specific business use case is settled there, on a few hundred real prompts, not on a public league table.
Limitation evaluation is a separate pass, and it eliminates rather than ranks. For each shortlisted model, write down the context-window ceiling, the maximum output tokens, the modalities it accepts and produces, whether it supports tool use and structured output, which regions offer it and whether a geo or global inference profile covers it, and whether it can be fine-tuned or served on Provisioned Throughput. Those answers are rarely the ones you assume: Nova Micro and Nova Pro both support client-side tool calling, neither supports structured outputs, and both cap output at 5K tokens however large the input window is. Any one of those can rule out the model that scored best on every benchmark, and finding out during integration is far more expensive than reading the model card first.
Measuring price-to-performance
Once two candidates are close, put a number on them. Run every candidate over the same golden set with the same prompt. Record the pass rate, the mean input and output token counts, and p50 and p99 latency. Multiply the token counts by each model’s published rates, divide the spend by the answers that passed, and you have a cost per correct answer. That figure makes a cheap model failing a fifth of the time comparable with an expensive one that rarely fails.
The per-token list price misleads on its own. A weaker model is often more verbose, so it bills more output tokens for the same answer, and a failure usually means a retry, which bills the whole call again. Its effective cost per useful answer can land above a dearer model’s, and the p99 column sits right next to the pass rate for anything on a latency-sensitive path.
Measure the ratio per query class rather than per workload. If the easy queries and the hard ones show much the same cost per correct answer on the small model, one model serves everything and routing logic adds nothing. When the small model’s ratio collapses on the hard class and holds on the easy one, tiering by query complexity is worth the routing it takes to build, and the measured split says where to draw the line. Batching the calls nobody is waiting on, and reserving throughput for the steady ones, sit on top of that choice rather than rescuing a model that loses on the ratio.
What we’ll filter on
- Modality, what goes in and what comes out? Text, image or video input, image or video generation, or embeddings.
- Reasoning difficulty, an easy snap judgement or a hard multi-step problem that justifies a large model?
- Cost and latency budget, how sensitive is this workload to per-token price and response time at volume?
- Context-window size, does the task feed in long documents and retrieved context, or almost nothing?
- Customisation, does it need fine-tuning to learn a style or vocabulary, or will prompting do?
- Region, access, and lifecycle, is the model offered where you need it, enabled for the account, and still Active?
The landscape
Amazon Nova (Micro, Lite, Pro). Amazon’s first-party text and multimodal family, tiered by size. Nova Micro is text-only, with a 128K window and the cheapest, fastest responses, suiting high-volume classification and extraction. Nova Lite and Nova Pro accept text, images and video over 300K windows, Lite as the balanced mid-tier and Pro as the most capable of the three. All three cap output at 5K tokens and support fine-tuning. Nova Premier and the first Nova Sonic are legacy, with end-of-life on 14 September 2026; Nova 2 Lite and Nova 2 Sonic are the active successors.
Amazon Nova Canvas and Nova Reel (legacy). The generation side of the Nova family: Canvas produced images from text prompts, Reel short video. Both reach end-of-life on 30 September 2026, so neither is a choice today.
Amazon Titan. Amazon’s earlier first-party line, now embeddings. Titan Text Embeddings V2 turns text into vectors for retrieval and semantic search over an 8K window, with configurable output dimensions, and Titan Multimodal Embeddings G1 does the same for text and images. Titan Image Generator G1 v2 is legacy and past its end-of-life date, and no Titan text-generation model remains.
Anthropic Claude. A general-purpose text and image-input family, frequently the pick for hard reasoning, nuanced writing, and tasks where output quality carries the feature. Several sizes are offered, so capability trades against cost within one provider, and individual versions move to legacy on their own schedules, so read the model card before pinning an ID.
Meta Llama. Open-weight text models, with image input on the 3.2 and 4 lines, across several sizes. Several are fine-tunable on Bedrock, which few third-party models are.
Mistral. Efficient text models spanning small, fast options through larger ones, often chosen for quality-per-cost balance on general text.
Cohere. An embeddings and reranking line on Bedrock rather than a text-generation one. Embed v4, Embed English and Embed Multilingual are active, while Command R and Command R+ have passed their end-of-life dates. That makes Cohere a candidate for the retrieval feature, not the chat one.
Stability AI. The text-to-image options are Stable Image Ultra for photorealistic output, Stable Diffusion 3.5 Large for high-volume creative assets, and Stable Image Core for the fast, cheap end. Thirteen further Stability image services cover editing and control work, from inpainting and background removal to sketch-to-image, and all of them take an input image.
Luma AI. Luma Ray 2 generates a 5 or 9 second clip at 540p or 720p from a text prompt, optionally keyframed on images you supply. It runs as an asynchronous job through StartAsyncInvoke and writes the MP4 to an S3 bucket you name.
AI21. Jamba 1.5 Large and Jamba 1.5 Mini both sit on the legacy list with an end-of-life date of 26 November 2026, so this provider is not a starting point for new work.
Two operational choices sit on top of the model pick. On-demand inference bills per token with no commitment, the default for variable traffic; Provisioned Throughput reserves model units at a fixed hourly price for steady, high-volume, latency-sensitive workloads, and Nova Micro, Lite and Pro support it. Cross-Region inference profiles, geo-scoped or global, let a request be served from any of several regions, raising available throughput without pinning a feature to one region’s limits.
Evaluation
Side by side
| Model class | Modality | Reasoning tier | Cost / latency | Context window | Fine-tuning | Typical fit |
|---|---|---|---|---|---|---|
| Nova Micro | Text in, text out | Easy | Lowest | 128K | ✓ | High-volume classify / extract |
| Nova Lite | Text, image, video in | Easy to medium | Low | 300K | ✓ | Balanced everyday text and vision |
| Nova Pro | Text, image, video in | Hard | Higher | 300K | ✓ | Harder reasoning with images |
| Claude (large) | Text and image in | Hard | Higher | Large | ✗ on current versions | Nuanced writing, hard reasoning |
| Llama / Mistral | Text, some image in | Easy to hard by size | Varies by size | Varies | Some Llama sizes | General text, cost-balanced |
| Titan / Cohere embeddings | Text in, vectors out | ✗ (not generative) | Low | 8K on Titan V2 | ✗ | Retrieval and semantic search |
| Stability AI | Text in, image out | ✗ | Per image | N/A | ✗ | Image generation |
| Luma Ray 2 | Text in, video out | ✗ | Per clip | N/A | ✗ | 5 or 9 second clips |
Reading the table against the four features: the classifier needs Nova Micro or a small text tier; the retrieval assistant takes an embeddings model, not a chat model at all; the hard reasoner justifies Nova Pro or a large Claude; the marketing tool needs Stability AI. One flagship chat model was the wrong answer for three of the four. Read the Cost / latency column as a shorthand for how each class behaves, not a list price; between two close candidates the figure that decides is the measured cost per correct answer over your own golden set.
Routing a workload to a model class
The decision is a small cascade: settle modality, then, for the text branch, let reasoning difficulty and volume choose the size.
The gates after the size pick are the ones teams forget, and any one of them can send you back a step.
The solution
The classifier is the clearest saving. Eight labels, one short input, one short output, running millions of times a month: this is an easy judgement at enormous volume, which is the profile Nova Micro is built for. Moving it off the flagship and onto the cheapest text tier cuts the per-call price and the latency at once, and because the task never needed deep reasoning, accuracy holds. At this volume the model choice is the single biggest lever on the bill. If the traffic is steady and heavy enough, this is also the feature where Provisioned Throughput and a cross-Region inference profile reserve capacity and spread load, so latency stays flat under peak.
The retrieval assistant needs a change of model class, not a smaller chat model. Finding the passages closest in meaning to a question is an embeddings job. Titan Text Embeddings V2 or a Cohere Embed model turns documents and queries into vectors, and the nearest vectors are the relevant passages. A chat model asked to “find similar text” is doing the wrong job expensively. The embeddings model produces the vectors that fill the index, and a separate generative model writes the answer from the retrieved passages, chosen by running the modality-and-difficulty cascade again. That split is the heart of a retrieval-augmented setup on Bedrock.
The hard reasoner is the one feature that genuinely needs a large model. Multi-step policy questions where a subtle error is costly are where Nova Pro or a large Claude changes the answer, not just the token count, and where a 300K window matters if the policy documents are long. Because it runs far less often than the classifier, its higher per-call cost barely moves the total, so this is the right place to spend.
The marketing tool needs the right modality. A text model cannot produce an image, so this routes to an image model, billed per image rather than per token, and today that means Stability AI: Stable Image Core for volume, Stable Image Ultra for the hero shots. Amazon’s own Canvas and Reel are on the legacy list, so modality narrows the field to the models that can produce a picture and lifecycle status decides which of them you can still call. If short video is ever on the brief, Luma Ray 2 covers that branch as an asynchronous job. Region and model access are the last gates, because generation models are not offered everywhere and, like every model on Bedrock, have to be enabled before the first call works.
Worked example
Put the two text features side by side and the sizing logic falls out. The classifier receives a short message, returns one of eight labels, and runs, say, three million times a month. The reasoner receives a policy question with a few supporting paragraphs, returns a careful multi-paragraph answer, and runs a few thousand times a month.
For the classifier, the honest first move is to ask whether it should be here at all. With a few thousand labelled tickets in the history, Amazon Comprehend custom classification or a small model trained on SageMaker does eight-way tagging for less again, deterministically, with an accuracy number you can watch. Assume this team has no labelled history yet and phrasings that keep shifting, so the foundation model holds the slot. The input and output are both tiny, so per-token price multiplied by volume is the whole story. Nova Micro across three million short calls costs dramatically less than the same calls on a flagship, and the answers are just as good, because eight-way tagging is not a reasoning problem. On-demand is fine to start, and if the volume stays high and steady, Provisioned Throughput plus a cross-Region profile keeps latency flat and unit cost down.
For the reasoner, the input carries real context and the output is long and must be right. A large model such as Nova Pro or a large Claude is the correct spend, and a 300K window matters because the policy paragraphs have to fit. The per-call cost is far higher, but a few thousand calls a month against millions for the classifier leaves the reasoner a rounding error on the bill. The team started with the opposite: one big model for both, overpaying on the feature that runs constantly to avoid re-deciding on the feature that barely runs at all.
What’s worth remembering
- Decide the model class before the model: modality first (text, image or video input, image or video generation, or embeddings), because the wrong class cannot produce the right output at any price.
- Before sizing a model for an easy judgement, ask whether it needs a foundation model at all; a fixed label set with labelled history belongs to a purpose-trained classifier, and the foundation model is the right answer only when that history or stability is missing.
- Retrieval is an embeddings job, not a chat job; use Titan Text Embeddings V2 or a Cohere Embed model to build the index, then a separate generative model to write the answer.
- Pick the smallest model that clears the quality bar; on easy, high-volume tasks a small model such as Nova Micro is both cheaper and faster, and accuracy holds because the task never needed reasoning.
- Region availability, model access, and lifecycle status are gates, not details: a model has to be offered in your region, enabled for the account, and still Active before any call succeeds, and a Legacy model closes to new accounts immediately and to everyone at its end-of-life date.