The situation
A travel booking company is adding three model-backed features in the same release. The three have almost nothing in common except the API they will call.
The first is a chat assistant on the booking site. It answers questions about existing bookings, cancellation windows and baggage rules in English, Japanese and German, because those are the three markets the company sells in. Around thirty thousand conversations a day, a person watching the reply appear on screen, and a three-thousand-token block of booking policy and tone instructions sent at the top of every turn of every conversation.
The second is an overnight job. Every supplier contract the commercial team signs, typically two hundred pages of PDF, gets turned into a one-page brief listing commission rates, blackout dates and termination terms. About sixty contracts a week, all processed between midnight and six, and nobody is waiting.
The third is accessibility work that has been outstanding for two years. Four hundred thousand hotel photographs in the catalogue have no alternative text, so a screen reader announces “image” and moves on. The feature looks at a photograph and writes a sentence describing what is in it.
The proposal on the table is one model for all three, chosen by taking the most capable thing in the account and pointing everything at it. Whether generative AI suits each of these jobs at all, and what the business gets from each, is a separate weighing, and the eight business factors work through that. This is the design-time version of the decision: the selection criteria you run down once the features are agreed and somebody has to name three models and choose the settings they are called with.
What actually matters
Start with modality, because it eliminates rather than ranks. Modality means what kind of data a model takes in and what kind it produces. Text in and text out covers the chat assistant and the contract summariser. The photo feature needs a model that accepts an image as input and answers in words. Reading a picture, drawing one, and encoding a passage as a vector are three separate capabilities, and no prompt moves a model between them. A model of the wrong modality is not a model that scores badly on this job. It cannot attempt it.
Multi-lingual support is the second eliminator, and it catches people out because it looks like a prompt problem. Asking a model to “reply in Japanese” does not add Japanese to a model that saw very little of it in training; it produces stilted, occasionally wrong Japanese with confident grammar. Language coverage is a property of the model, and the published language list is where you start rather than where you finish. A German speaker and a Japanese speaker have to read fifty real answers each and say whether they would send them to a customer.
Then input/output length, measured against the model’s context window. Two hundred pages of contract is roughly a hundred thousand words. A working figure for English is that a token runs to about three quarters of a word, which puts the document near a hundred and thirty thousand tokens before any instructions are added. The context window has to hold the instructions, the document, and the room the answer will occupy, all at once. The output side has its own separate ceiling, usually far smaller than the input one. What goes into that window, and in what proportions, is a budgeting exercise in itself; here it is a gate. A document that does not fit means either a longer-context model or splitting the contract into pieces and summarising the summaries.
Model size and model complexity are what remains once the gates have run, and they trade against latency and cost in a fairly predictable way. A larger, more complex model handles ambiguity and multi-step reasoning better, takes longer to answer, and charges more per token. A smaller one is quicker and cheaper and gives up ground on the hard cases. Which of those matters depends entirely on who is waiting. Latency is a real constraint for the chat assistant, where someone is watching a cursor blink, and close to irrelevant for a job that runs at two in the morning. Cost is charged per thousand input tokens and per thousand output tokens, at different rates for each. A feature’s monthly bill is its volume multiplied by its token counts multiplied by those rates, and the arithmetic has some surprises in it once repeated context is included.
Two more criteria are properties of the model and the platform rather than of the job. Prompt caching lets a provider keep the processed form of a long, unchanging block at the start of a prompt and reuse it on the next call. You are billed a reduced rate for the cached part, and the first token comes back sooner. The chat assistant sends the same three thousand tokens of policy on every turn, which is exactly the shape prompt caching was built for, and the saving grows with how repetitive the prefix is. It requires model support and an identical prefix, so a system prompt with the current time interpolated into the first line caches nothing. Customization is the other one: whether the model can be fine-tuned or distilled at all, which several models in any catalogue cannot. None of these three features needs it today, and a model that forecloses it is a narrower choice than one that does not. How far to customise is its own decision, and it starts with whether the option exists.
What we’ll filter on
- Modality. Which of the four shapes is this: text to text, image to text, text to image, or text to vector?
- Multi-lingual coverage. Which languages does the model genuinely handle to a standard a customer would accept, verified by someone who speaks them?
- Input/output length. Does the longest input fit the context window alongside the instructions, and does the longest answer fit the output ceiling?
- Model size and model complexity. How much reasoning capacity does the task actually consume, and does a smaller tier clear the quality bar?
- Latency. Is a person waiting on the answer, or does the work run unattended?
- Cost and platform support. What are the per-thousand-token rates in and out, does the model support prompt caching for a repeated prefix, and does it support customization if that is ever wanted?
The landscape
Amazon Bedrock is the managed service that puts a catalogue of foundation models from several providers behind one API, so switching between them is a change of model identifier rather than a change of integration. Read the catalogue by class rather than by vendor, because the classes are what the criteria above sort on, and the vendor names change faster than the classes do. The same catalogue read family by family fills in which provider sits where. Where Bedrock sits among the AWS building blocks is a separate question; this is what sits inside it.
Small, fast text models
Text in, text out, at the cheapest rate and the lowest latency in the catalogue. They summarise, classify, extract fields, reformat and answer from supplied context perfectly well. They lose ground on multi-step reasoning, on long chains of instructions, and often on languages other than English. This class is where a high-volume feature should start, not where it should be assumed to fail.
General-purpose and flagship text models
The larger tiers. More model complexity, better handling of ambiguity, longer instruction chains followed reliably, and stronger performance across languages. They cost several times more per token and take longer to answer. The long-context members of this class are the ones with context windows measured in the hundreds of thousands of tokens, which is what a two-hundred-page document needs.
Multimodal models
They accept an image (sometimes video or audio) alongside text and answer in text. Describing a photograph, reading a scanned form, or answering a question about a chart all live here. Modality is the filter that puts a job in this class, and once it is here the smaller tiers are still worth testing first.
Image generation models
Text in, image out. Marketing imagery, product mockups, variations on an existing picture. None of the three features needs one, and it is worth naming the class so that “the model looks at a photo” and “the model makes a photo” do not get filed together.
Embedding models
Text or an image in, a vector out. They generate nothing a person reads. They are the component that makes semantic search and retrieval work, they are cheap, and they are not an alternative to a text model.
Amazon Nova and Amazon SageMaker JumpStart
Amazon Nova is the AWS first-party family in the Bedrock catalogue, and it spans several of these classes at once. There are text tiers running from a small, fast model up to a flagship, multimodal tiers that accept images and video, and separate models for image and video generation. Nova is usually the cheapest credible starting point for a text or multimodal job on Bedrock, which makes it a sensible baseline to measure the alternatives against.
Amazon SageMaker JumpStart is the other route to a model. It is a hub of open-source pre-trained models inside Amazon SageMaker AI, and deploying one from it gives you a SageMaker endpoint in your own account running those weights. That changes what you own: an instance billed by the hour whether requests arrive or not, an autoscaling configuration, and a patching schedule. In exchange you get control over the machine and the ability to run a model Bedrock does not carry. For three features that have no such requirement, JumpStart is a route to know about and not to take.
Evaluation
Side by side
| Model class | Text in, text out | Image understood | Long context | Latency | Cost per token | Customization offered |
|---|---|---|---|---|---|---|
| Small, fast text (Nova Micro tier) | ✓ | ✗ | ✗ | lowest | lowest | usually ✓ |
| General-purpose text | ✓ | ✗ | some | moderate | moderate | often ✓ |
| Flagship, long context | ✓ | usually | ✓ | highest | highest | often ✗ |
| Multimodal | ✓ | ✓ | some | moderate | moderate to high | sometimes ✓ |
| Image generation | ✗ | ✗ | ✗ | seconds per image | per image | rarely |
| Embedding | ✗ | ✗ | ✗ | very low | very low | rarely |
The first three columns answer yes or no, so they remove candidates. The last three answer with a number or a rate, so they order whatever survived. Running them in that order is what stops a shortlist becoming a ranking of the whole catalogue: capability decides who is eligible, and the rates decide between the eligible.
Reading the table against three jobs
The chat assistant clears the modality gate everywhere, so it is decided by latency, by multi-lingual quality and by cost at thirty thousand conversations a day. That combination points down the table rather than up it, with one condition attached. The small tier has to hold up in Japanese and German, and that is the criterion most likely to push this feature a tier higher than an English-only version of the same job would need.
The contract summariser fails the long-context column on most of the table, and that single filter does most of the work. Nobody is waiting at two in the morning, so the latency column has nothing to say. Cost is dominated by input rather than output, because a hundred and thirty thousand tokens go in and about a thousand come out.
The photo describer fails the modality gate on every class except multimodal, which leaves one row and then a choice of tier inside it. Four hundred thousand images at one sentence each is a large number of small outputs, so the per-token rate matters more than anything else once a tier is good enough.
The solution
Three features, three models, one internal interface in front of them so that changing any of the three is a configuration change.
The chat assistant runs on a small, fast text model, with the three-thousand-token policy block turned into a cached prefix using prompt caching. The prefix has to be byte-identical between calls for the cache to hit, so anything variable, the customer’s name, the current date, the booking reference, goes after the cached block rather than inside it. That cuts both the bill and the time to the first token on every turn after the first. Verify the multi-lingual side before this ships: fifty real German and fifty real Japanese conversations, read by people who speak them, scored for whether the answer is correct and whether the register suits a customer. If the small tier fails on Japanese and passes on English and German, there are two options: a larger model for every language, or routing by language. The second is more machinery than most teams want on day one.
The contract summariser runs on a long-context model, because input/output length is the criterion that decides it and nothing else comes close. Confirm the arithmetic against the longest contract in the archive rather than the average one. Set the output ceiling to fit a one-page brief with room to spare. Latency does not enter the decision at all, which leaves the choice to be made on context window and per-token cost alone. The overnight window also makes batch invocation rather than a live call worth considering for the discount.
The photo describer runs on a multimodal model, starting with the cheapest tier that produces usable sentences. Score it on two hundred real photographs against alt text a human has written. Check for the failure that matters in this domain: a description that invents a sea view, a swimming pool or a wheelchair ramp that is not in the picture. Four hundred thousand images is a backfill rather than a live feature, so run it as a batch and keep the per-image cost in front of you. Write the prompt to describe what is visible instead of inferring what the room is like.
Setting the inference parameters
Inference parameters are set per request rather than per model, and they change how the model turns its predictions into text. They are not training settings, and confusing the two is a common slip: epochs, learning rate and batch size belong to training, while these belong to every individual call.
Temperature controls how much randomness there is in the choice of the next token. Near zero, the model takes the most probable option almost every time and the output is repetitive and predictable. Higher, and it will take less likely options, which produces more varied and more creative writing and more opportunities to go wrong. Extraction, classification and structured summarising want temperature near zero. Drafting marketing copy wants it higher.
Top-p and top-k cut the pool of tokens the model is allowed to sample from. Top-k limits the choice to the k most likely next tokens; top-p limits it to the smallest set of tokens whose probabilities add up to p. Temperature rescales the probabilities and these two truncate the result, so moving all three at once produces behaviour nobody can reason about; tuning one at a time is enough for almost every application.
Max tokens caps the length of the response, which caps the output half of the bill. Stop sequences end generation when a particular string appears. That is how a response finishes cleanly at the end of a JSON object, or before the model starts a new section nobody asked for.
Two gotchas are worth stating plainly. Max tokens truncates; it does not summarise. Setting it to five hundred on a job that wants a thousand gives you the first five hundred tokens and a sentence cut in half. Ask for a shorter answer in the prompt, and keep max tokens as the ceiling behind it. And a low temperature reduces variation without guaranteeing identical output, so a requirement that the same input always produces the same output is not met by turning temperature down.
Against the three features: temperature near zero for the contract briefs, because commission rates and blackout dates are extraction and a creative rendering of a termination clause is a liability. Low for the chat assistant, with max tokens set to keep answers to a few sentences and a stop sequence guarding the format. Low for the photo descriptions too, with a tight max tokens, because one accurate sentence is the deliverable and a paragraph of atmosphere is not.
Worked example
Take the contract summariser and run the criteria in order.
Modality removes the image and embedding classes at once. Text goes in, text comes out. The contracts are PDFs, so something has to extract their text before the model sees it, and that is a document-processing step in front of the model rather than a model choice.
Multi-lingual turns out to matter after all, because a fifth of the supplier contracts are in German. That does not change the class, but it does change which model inside the class, and it adds twenty German contracts to the scored example set.
Input/output length is the gate that decides. The longest contract in the archive extracts to about a hundred and eighty thousand tokens. Add two thousand tokens of instructions and the format description, then leave room for a thousand-token brief, and the model needs a context window comfortably above two hundred thousand tokens. Anything smaller means splitting each contract into sections, summarising each section, then summarising those summaries, which is more machinery and more places to lose a blackout date.
Model size and model complexity favour a larger model here. Reading two hundred pages and finding every clause that changes a commission rate is exactly the multi-step work the smaller tiers give up on, and the scored example set will show it.
Latency is not a filter for this feature. Six hours of window for sixty contracts a week means a model taking ninety seconds per document is fine.
Cost is the last check, and it is smaller than it looks. Sixty contracts a week at a hundred and eighty thousand input tokens is under fifty million input tokens a month, with output in the tens of thousands. Prompt caching does not help, because every contract is a different document and there is no repeated prefix beyond the instructions. Customization does not enter it either; the instructions and a handful of worked examples in the prompt do the job without touching the model.
What’s worth remembering
- Modality and multi-lingual coverage are gates that remove candidates, while model size, model complexity, latency and cost are rates that order the survivors, so run the gates first.
- Input/output length is checked against the longest real input plus the instructions plus the room the answer needs, and the output ceiling is a separate, much smaller limit than the context window.
- Prompt caching pays off when a long block at the start of the prompt is identical between calls, so keep anything variable out of the cached prefix.
- Amazon Bedrock puts a catalogue of models behind one API and Amazon Nova is the AWS first-party family within it; Amazon SageMaker JumpStart is the other route, for when you want the weights on an endpoint you own and operate.
- Inference parameters are per request: temperature and top-p control randomness, top-k limits the candidate pool, max tokens caps length and therefore output cost, and stop sequences end a response cleanly.
- Max tokens truncates rather than summarises, and a low temperature reduces variation without making output identical, so ask for brevity in the prompt and put any real determinism requirement in code.