The situation
A travel booking company is adding three model-backed features in the same release. The three have almost nothing in common except the API they will call.
The first is a chat assistant on the booking site. It answers questions about existing bookings, cancellation windows and baggage rules in English, Japanese and German, because those are the three markets the company sells in. Around thirty thousand conversations a day, a person watching the reply appear on screen, and a three-thousand-token block of booking policy and tone instructions sent at the top of every turn of every conversation.
The second is an overnight job. Every supplier contract the commercial team signs, typically two hundred pages of PDF, gets turned into a one-page brief listing commission rates, blackout dates and termination terms. About sixty contracts a week, all processed between midnight and six, and nobody is waiting.
The third is accessibility work that has been outstanding for two years. Four hundred thousand hotel photographs in the catalogue have no alternative text, so a screen reader announces “image” and moves on. The feature looks at a photograph and writes a sentence describing what is in it.
The proposal on the table is one model for all three, chosen by taking the most capable thing in the account and pointing everything at it. The same instinct run against a wider set of filters, compliance and Regional availability among them, is a separate walk-through. This is the design-time version of the decision: the selection criteria you run down once the features are agreed and somebody has to name three models and choose the settings they are called with.
What actually matters
Start with modality, because it eliminates rather than ranks. Modality means what kind of data a model takes in and what kind it produces. Text in and text out covers the chat assistant and the contract summariser. The photo feature needs a model that accepts an image as input and answers in words. Reading a picture, drawing one, and encoding a passage as a vector are three separate capabilities, and no prompt moves a model between them. A model of the wrong modality does not score badly on this job. It cannot produce that output at all.
Multi-lingual support is the second eliminator, and it catches people out because it looks like a prompt problem. Asking a model to “reply in Japanese” does not add Japanese to a model whose training data held very little of it; the output is stilted and occasionally wrong. Language coverage is a property of the model, and the published list is where you start rather than where you finish. Amazon Nova, for instance, is documented as supporting over two hundred languages while naming fifteen it is optimised for, English, German and Japanese among them, and that gap is where the trouble sits. So a German speaker and a Japanese speaker have to read fifty real answers each and say whether they would send them to a customer.
Then input/output length, measured against the model’s context window. Two hundred pages of contract is roughly a hundred thousand words. A common rule of thumb puts an English token at about three quarters of a word, landing the document near a hundred and thirty thousand tokens, but tokenisation is model-specific and the rule is a first estimate only. Bedrock’s CountTokens API returns the exact input-token count a named model would be billed for. The context window has to hold the instructions, the document, and the room the answer will occupy, all at once. The output side has its own separate ceiling, usually far smaller than the input one. What goes into that window, and in what proportions, is a budgeting exercise in itself; here it is a gate. A document that does not fit means either a longer-context model or splitting the contract into pieces and summarising the summaries.
Model size and model complexity are what remains once the gates have run, and they trade against latency and cost in a fairly predictable way. A larger, more complex model handles ambiguity and multi-step reasoning better, takes longer to answer, and costs more per token. A smaller one answers quicker, costs less, and is less accurate on the hard cases. Which of those matters depends entirely on who is waiting. Latency is a real constraint for the chat assistant, where someone is watching a cursor blink, and close to irrelevant for a job that runs at two in the morning. Cost is charged per million input tokens and per million output tokens, at different rates for each, and per million is the unit the Bedrock price list quotes in. A feature’s monthly bill is its volume multiplied by its token counts multiplied by those rates, and the arithmetic has some surprises in it once repeated context is included.
Two more criteria are properties of the model and the platform rather than of the job. Prompt caching lets Bedrock keep the processed form of a long, unchanging block at the start of a prompt and reuse it on the next call. Tokens read from the cache are billed at the model’s cache-read rate, and time to the first token drops. Tokens written to the cache can be billed above the standard input rate, so a prefix rewritten more often than it is reused costs more than it saves. The chat assistant sends the same three thousand tokens of policy on every turn, and the saving grows with how repetitive the prefix is. Three constraints come with it. The prefix has to be identical, so a system prompt with the current time in its first line misses on every call. Where the model takes explicit cache checkpoints, each one has a model-specific minimum between 512 and 4,096 tokens, so a three-thousand-token block clears some and not others; Amazon Nova also caches eligible text prefixes implicitly, with no checkpoint in the request. And caching applies to on-demand inference only, never to batch jobs.
Customization is the other one. Bedrock offers supervised fine-tuning, reinforcement fine-tuning and distillation, and support varies by model. Amazon Nova Premier cannot be fine-tuned on Bedrock at all, though it can act as the teacher that distils a smaller Nova. None of these three features needs customization today, and a model that forecloses it is a narrower choice than one that does not. How far to customise is its own decision, and it starts with whether the option exists.
What we’ll filter on
- Modality. Which of the four shapes is this: text to text, image to text, text to image, or text to vector?
- Multi-lingual coverage. Which languages does the model genuinely handle to a standard a customer would accept, verified by someone who speaks them?
- Input/output length. Does the longest input fit the context window alongside the instructions, and does the longest answer fit the output ceiling?
- Model size and model complexity. How much reasoning capacity does the task actually consume, and does a smaller tier clear the quality bar?
- Latency. Is a person waiting on the answer, or does the work run unattended?
- Cost and platform support. What are the per-million-token rates in and out, does the model support prompt caching for a repeated prefix, and does it support customization if that is ever wanted?
The landscape
Amazon Bedrock is the managed service that puts a catalogue of foundation models from several providers behind one API, so switching between them is a change of model identifier rather than a change of integration. Read the catalogue by class rather than by vendor, because the classes are what the criteria above sort on, and the vendor names change faster than the classes do. The same catalogue read family by family fills in which provider sits where. Where Bedrock sits among the AWS building blocks is a separate question; this is what sits inside it.
Small, fast text models
Text in, text out, at the cheapest rate and the lowest latency in the catalogue. They summarise, classify, extract fields, reformat and answer from supplied context perfectly well. They lose ground on multi-step reasoning, on long chains of instructions, and often on languages other than English. This class is where a high-volume feature should start, not where it should be assumed to fail.
General-purpose and flagship text models
The larger tiers. More model complexity, better handling of ambiguity, longer instruction chains followed reliably, and stronger performance across languages. They cost several times more per token and take longer to answer. The long-context members of this class are the ones with context windows measured in the hundreds of thousands of tokens, which is what a two-hundred-page document needs.
Multimodal models
They accept an image (sometimes video or audio) alongside text and answer in text. Describing a photograph, reading a scanned form, or answering a question about a chart all live here. Modality is the filter that puts a job in this class.
Image generation models
Text in, image out. Marketing imagery, product mockups, variations on an existing picture. None of the three features needs one, and it is worth naming the class so that “the model looks at a photo” and “the model makes a photo” do not get filed together.
Embedding models
Text or an image in, a vector out. They generate nothing a person reads. They are the component that makes semantic search and retrieval work, they are cheap, and they are not an alternative to a text model.
Amazon Nova and Amazon SageMaker JumpStart
Amazon Nova is the AWS first-party family in the Bedrock catalogue, and it spans several of these classes at once. Nova Micro is text only, with a 128K context window. Nova Lite, Pro and Premier take text, images, video and documents and answer in text, with context windows of 300K for Lite and Pro and 1M for Premier, against a documented maximum output of 10K tokens. Nova Canvas generates images, Nova Reel generates video, Nova Sonic handles speech. A second generation now sits alongside the first: Nova 2 Lite, Nova 2 Sonic and Nova Multimodal Embeddings, documented at up to 1M tokens of context and up to 65,536 tokens in a single response. AWS describes Micro as its lowest-latency model at very low cost, which makes it a sensible baseline to measure alternatives against.
Amazon SageMaker JumpStart is the other route to a model. It is a hub of pretrained models inside Amazon SageMaker AI, from both proprietary and publicly available providers, and deploying one gives you a SageMaker endpoint in your own account running those weights. That changes what you own: an instance billed by the hour whether requests arrive or not, an autoscaling configuration, and a patching schedule. You get control over the machine and can run a model Bedrock does not carry. For three features that have no such requirement, JumpStart is a route to know about and not to take.
Evaluation
Side by side
| Model class | Text in, text out | Image understood | Long context | Latency | Cost per token | Customization offered |
|---|---|---|---|---|---|---|
| Small, fast text (Nova Micro tier) | ✓ | ✗ | ✗ | lowest | lowest | usually ✓ |
| General-purpose text | ✓ | ✗ | some | moderate | moderate | often ✓ |
| Flagship, long context | ✓ | usually | ✓ | highest | highest | often ✗ |
| Multimodal | ✓ | ✓ | some | moderate | moderate to high | sometimes ✓ |
| Image generation | ✗ | ✗ | ✗ | seconds per image | per image | rarely |
| Embedding | ✗ | ✗ | ✗ | very low | very low | rarely |
The first three columns answer yes or no, so they remove candidates. The last three answer with a number or a rate, so they order whatever survived. Running them in that order stops a shortlist becoming a ranking of the whole catalogue.
Reading the table against three jobs
The chat assistant clears the modality gate everywhere, so it is decided by latency, by multi-lingual quality and by cost at thirty thousand conversations a day. That combination points down the table rather than up it, with one condition attached. The small tier has to hold up in Japanese and German, and that is the criterion most likely to push this feature a tier higher than an English-only version of the same job would need.
The contract summariser fails the long-context column on most of the table, and that single filter does most of the work. Nobody is waiting at two in the morning, so the latency column drops out of the decision. Cost is dominated by input rather than output, because well over a hundred thousand tokens go in and about a thousand come out.
The photo describer fails the modality gate on every class except multimodal, which leaves one row and then a choice of tier inside it. Four hundred thousand images at one sentence each is a large number of small outputs, so the per-token rate matters more than anything else once a tier is good enough.
The solution
Three features, three models, one internal interface in front of them so that changing any of the three is a configuration change.
The chat assistant runs on a small, fast text model, with the three-thousand-token policy block set up as a cached prefix. Anything variable, the customer’s name, the current date, the booking reference, goes after the cached block rather than inside it. Check the chosen model’s minimum checkpoint size first: three thousand tokens clears a 512 or 1,024 minimum and falls short of 4,096. Verify the multi-lingual side before this ships: fifty real German and fifty real Japanese conversations, read by people who speak them, scored for whether the answer is correct and whether the register suits a customer. If the small tier fails on Japanese and passes on English and German, there are two options: a larger model for every language, or routing by language.
The contract summariser runs on a long-context model, because input/output length is the criterion that decides it and nothing else comes close. Confirm the arithmetic against the longest contract in the archive rather than the average one. Set the output ceiling to fit a one-page brief with room to spare. Latency does not enter the decision at all, which leaves the choice to be made on context window and per-token cost alone. The overnight window also suits batch inference, which Bedrock prices at half the on-demand rate. Batch does not support tool calling or structured output, so the brief’s format has to come from the prompt rather than a response schema.
The photo describer runs on a multimodal model, starting with the cheapest tier that produces usable sentences. Score it on two hundred real photographs against alt text a human has written. Check for the failure that matters in this domain: a description naming a sea view, a swimming pool or a wheelchair ramp that is not in the picture. Four hundred thousand images is a backfill rather than a live feature, so run it in batch rather than on demand, and keep the per-image cost in front of you. Write the prompt to describe what is visible rather than to characterise the room.
Setting the inference parameters
Inference parameters are set per request rather than per model, and they change how the model turns its predictions into text. They are not training settings, and confusing the two is a common slip: epochs, learning rate and batch size belong to training, while these belong to every individual call.
Temperature reshapes the probability distribution the next token is sampled from. Near zero, the distribution steepens, the highest-probability token is selected almost every time, and the output is repetitive and predictable. Higher, the distribution flattens and lower-probability tokens get selected more often, which produces more varied writing and more errors. Extraction, classification and structured summarising call for temperature near zero. Drafting marketing copy calls for it higher.
Top-p and top-k cut the pool of tokens sampled from. Top-k limits the pool to the k most likely next tokens; top-p limits it to the smallest set of tokens whose probabilities add up to p. Temperature rescales the probabilities and these two truncate the result, so moving all three at once produces behaviour nobody can reason about; tuning one at a time is enough for almost every application.
Max tokens caps the length of the response, which caps the output half of the bill. Stop sequences end generation when a particular string appears. That is how a response finishes cleanly at the end of a JSON object, or before it runs on into a new section.
Two gotchas. Max tokens truncates; it does not summarise. Setting it to five hundred on a job that needs a thousand gives you the first five hundred tokens and a sentence cut in half. Ask for a shorter answer in the prompt, and keep max tokens as the ceiling behind it. And a low temperature reduces variation without guaranteeing identical output, so a requirement that the same input always produces the same output is not met by turning temperature down.
Against the three features: temperature near zero for the contract briefs, because commission rates and blackout dates are extraction and a creative rendering of a termination clause is a liability. Low for the chat assistant, with max tokens set to keep answers to a few sentences and a stop sequence to close the format. Low for the photo descriptions too, with a tight max tokens, because one accurate sentence is the deliverable and a paragraph of atmosphere is not.
Worked example
Take the contract summariser and run the criteria in order.
Modality removes the image and embedding classes at once. Text goes in, text comes out. The contracts are PDFs, so something has to extract their text before the model sees it, and that is a document-processing step in front of the model rather than a model choice.
Multi-lingual turns out to matter after all, because a fifth of the supplier contracts are in German. That does not change the class, but it does change which model inside the class, and it adds twenty German contracts to the scored example set.
Input/output length is the gate that decides. The longest contract in the archive extracts to about a hundred and eighty thousand tokens. Add two thousand tokens of instructions and the format description, then leave room for a thousand-token brief, and the model needs a context window comfortably above two hundred thousand tokens. Anything smaller means splitting each contract into sections, summarising each section, then summarising those summaries, which is more machinery and more places to lose a blackout date.
Model size and model complexity favour a larger model here. Reading two hundred pages and finding every clause that changes a commission rate is the multi-step work the smaller tiers handle least reliably, and the scored example set will show it.
Latency is not a filter for this feature. Six hours of window for sixty contracts a week means a model taking ninety seconds per document is fine.
Cost is the last check, and it is smaller than it looks. Sixty contracts a week at a hundred and eighty thousand input tokens is under fifty million input tokens a month, with output in the tens of thousands. Prompt caching does not help, because every contract is a different document and there is no repeated prefix beyond the instructions. Customization does not enter it either; the instructions and a handful of worked examples in the prompt do the job without touching the model.
What’s worth remembering
- Modality and multi-lingual coverage are gates that remove candidates, while model size, model complexity, latency and cost are rates that order the survivors, so run the gates first.
- Input/output length is checked against the longest real input plus the instructions plus the room the answer needs, and the output ceiling is a separate, much smaller limit than the context window.
- Prompt caching cuts input cost and first-token latency when a long block at the start of the prompt is identical between calls, so keep anything variable out of the cached prefix and check the model’s minimum checkpoint size.
- Amazon Bedrock puts a catalogue of models behind one API and Amazon Nova is the AWS first-party family within it; Amazon SageMaker JumpStart is the other route, for when you want the weights on an endpoint you own and operate.
- Inference parameters are per request: temperature reshapes the probability distribution, top-p and top-k both narrow the candidate pool, max tokens caps length and therefore output cost, and stop sequences end a response cleanly.
- Max tokens truncates rather than summarises, and a low temperature reduces variation without making output identical, so ask for brevity in the prompt and put any real determinism requirement in code.