Cheat Sheet: Model Selection and Inference

A fast revision sheet for model choice and inference options across Amazon Bedrock and self-hosted SageMaker.

Services at a glance

Decision rules

• If the task is text-only and high-volume, start at Nova Micro and only move up if quality falls short.
• If input includes images or video, you need a multimodal model (Nova Lite/Pro, Claude, others), not Micro or Titan text.
• If you need to generate images, reach for Stability; for video, Luma Ray 2. Nova Canvas and Nova Reel are legacy, reach end of life on 30 September 2026, and are not a choice for new work.
• If you need embeddings for RAG, reach for Titan Embeddings or Cohere Embed, not a chat model.
• If you are narrowing a shortlist, published performance benchmarks rank models against someone else’s tasks, so use them to cut the field; a Bedrock evaluation job over your own golden setGolden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions. settles which one ships.
• Before any score is compared, run the limitation evaluation and drop the models that cannot do the job at all: context window, maximum output tokens, modalities, tool use, supported languages, region and cross-region availability, and whether the model supports fine-tuning and Provisioned Throughput.
• If traffic is spiky and low commitment, use on-demand.
• If volume is steady and high, price out Provisioned ThroughputProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. against on-demand.
• If you fine-tune a model on Bedrock, you must serve it on Provisioned Throughput; an imported model is different, billing by the Custom Model Units it occupies in five-minute windows and scaling to zero when idle.
• If the job is large and offline with no latency need, use batch inferenceBatch inferenceSubmitting a bulk job of model calls to run asynchronously at a lower per-token price, trading immediacy for cost. for the discount.
• If a single region’s throughput quota is the limit, enable a cross-region inferenceCross-region inferenceLetting a request be served from any of several regions, raising effective throughput and riding out pressure in one of them. profile.
• If data must stay in one geography, scope the inference profile to same-geography regions and check residency.
• If the model is not on Bedrock, host it on SageMaker.
• If a SageMaker endpoint sits idle between bursts, use serverless to scale to zero and accept cold starts.
• If payloads are large or processing is slow, use SageMaker asynchronous inference with its queue.
• If you score a whole dataset once with no live endpoint, use SageMaker batch transform.
• If you host many models on shared infrastructure, use inference components for per-model resources and scaling, or a multi-model endpoint for a long tail of similar models sharing one container.
• If requests vary in difficulty, put Intelligent Prompt Routing in front and let it pick the cheapest tier that clears the bar.
• If failures are transient and sparse, retry with exponential backoff and jitter; if the dependency is failing most calls, open a circuit breaker and shed load until a probe says it has recovered.
• If you run a circuit breaker, keep its state in a shared store such as DynamoDB, so every Lambda invocation and every Step Functions execution reads the same state; state held in process memory resets on the next cold start and each concurrent worker trips independently.

Traps

• Nova Micro is text-only; picking it for an image or video task is wrong.
• Nova Canvas and Nova Reel are on the legacy list; an account with no recent history of calling them is refused access, and both are withdrawn entirely on 30 September 2026, so do not design new work around either.
• Video generation is asynchronous: Ray 2 runs through StartAsyncInvoke and delivers to S3, so there is no synchronous response to wait on.
• Provisioned Throughput is not just a cost lever; for some customised models it is the only serving path, not an optimisation.
• Batch inference is asynchronous and offline; it does not lower latency for live requests.
• Cross-region inference profiles move data across regions, which can breach a data-residency requirement.
• On-demand is quota-bound; hitting throttling means raising quota or moving to Provisioned Throughput, not retrying blindly.
• SageMaker serverless has cold starts; do not pick it when consistent low latency matters.
• Enabling model access in the console is a prerequisite; a model you have not enabled will not answer.
• Model availability differs by region; a model in one region may be absent in another.
• Bigger is not automatically better; the smallest model that clears the quality bar is usually the right pick on cost and latency.
• SageMaker batch transform has no persistent endpoint; do not use it for real-time serving.
• Asynchronous inference is for large payloads and long jobs, not a substitute for provisioned real-time throughput.
• Fine-tuning can change how you serve, and which way it lands depends on the base model: a custom Nova or Llama 3.3 70B goes on demand, a custom Llama 3.1 8B cannot.
• A hardcoded model id looks like a one-line change and behaves like an eight-deploy one once eight services call Bedrock: eight code edits, eight reviews, eight releases, and a window where half the fleet is on each model. Held in AppConfig, the swap is one configuration deployment with a bake time and an automatic rollback.
• Flattening per-model inference parameters into one shared blob breaks the models that do not share them; the parameter names and ranges differ by provider, so an indirection layer has to carry an inference configuration per model, not one global set.

Say it in one line

1. Nova Micro is text-only; Lite and Pro are multimodal; Canvas and Reel are legacy, so images come from Stability AI and video from Luma Ray 2.
2. Titan and Cohere give you embeddings; use them for RAG and search, not chat.
3. On-demand charges per token with no commitment and is bound by service quotas.
4. Provisioned Throughput reserves model unitsModel unitThe billing block Provisioned Throughput is sold in – one unit delivers a fixed tokens-per-minute rate for a specific model. per hour and is mandatory for some customised models, though not for one that offers on-demand custom serving, and not for an imported model.
5. Batch inference is async S3-to-S3, roughly half price, and offline only.
6. Cross-region inference profiles raise throughput by spreading load and can carry data-residency implications.
7. Enabling model access and checking regional availability are prerequisites before any call.
8. SageMaker real-time serves low-latency traffic on an always-on endpoint.
9. SageMaker serverless scales to zero and trades cold starts for idle savings.
10. SageMaker asynchronous inference queues large payloads and long-running jobs.
11. SageMaker batch transform scores a dataset offline with no persistent endpoint.
12. Inference components give each model on a shared endpoint its own CPU, memory, accelerators, copy count, and scaling (to zero copies); multi-model endpoints instead share one container, loading models into memory on demand at the cost of a cold-start penalty and a same-framework requirement.
13. Intelligent Prompt Routing routes within a family to the cheapest tier that meets a quality bar.
14. Pick the smallest model that clears the quality bar; move up only when it does not.