Cheat Sheet: Evaluation, Cost, and Operations

A fast revision pass over evaluating, monitoring, costing, and operating a generative AI app on Bedrock. Skim the tables, drill the decision rules, watch the traps.

Levers at a glance

Decision rules

• If you need one number to compare model changes, then run a Bedrock programmatic evaluation job against a fixed golden setGolden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions..
• If quality is fuzzy and subjective, then use LLM-as-a-judgeLLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. for scale, and sample to human review for the final word.
• If the app retrieves documents, then run a retrieve-and-generate RAG evaluation and read faithfulness and correctness separately.
• If the answer is well-written but states facts that are not in the retrieved text, then faithfulness is failing, not correctness.
• If the answer is grounded in the context but the context is wrong, then correctness is failing, not faithfulness.
• If you need labelled data or structured human ratings, then create a private work team and hold it to a written rubric.
• If some live responses must be checked by a person, then route them through a review loop built on Step Functions or SQS with a reviewer UI you own.
• If you can’t see what the model was sent, then enable Bedrock model invocation logging to S3 or CloudWatch first.
• If an agent gives a wrong answer, then read its trace to find which tool call or retrieval went wrong before touching the prompt.
• If volume is steady and high, then buy Provisioned ThroughputProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not.; if it is spiky, stay on-demand.
• If the work is offline and can wait, then use batch inference at half the on-demand rate.
• If a long system prompt or document repeats every call, then turn on prompt caching.
• If identical or near-identical prompts recur, then add response or semantic cachingSemantic cachingServing a cached answer when a new question is close enough in embedding space to one you’ve already answered. in front of the model.
• If two models in one family differ in cost, then put an Intelligent Prompt Routing endpoint in front of them. It predicts response quality per request, and it is tuned for English prompts.
• If the bill is a mystery, then apply cost allocation tags and application inference profilesInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code., read it in Cost Explorer, and alert with AWS Budgets.
• If spend climbs in a way no budget threshold would catch, then turn on AWS Cost Anomaly Detection and let the learned baseline flag the drift.
• If first-token feel matters, then stream the response and alarm on TimeToFirstToken, not just InvocationLatency.

Traps

• Provisioned Throughput is billed by the hour whether or not you send traffic; idle reserved capacity still costs money. Inference profiles don’t support it, so cross-Region routing and reserved capacity are separate choices.
• Batch inference is half price and asynchronous. It also drops tool calling, structured output and prompt caching, so it isn’t a drop-in for the online path.
• Faithfulness and correctness are different axes. A grounded answer can still be wrong, and a correct answer can still be unfaithful to bad context; don’t collapse them into one score.
• A judge model is cheap and consistent, and it carries its own errors into every score it produces. Anchor it to human review rather than treating it as ground truth.
• SageMaker Ground Truth, Amazon A2I and Mechanical Turk are closed to new customers. Existing customers can carry on as normal, and Bedrock still runs human evaluation jobs against a private work team of up to 50 workers, but a production review loop is something you assemble.
• A golden set of easy, in-scope cases misses regressions. Include hard cases, and out-of-scope prompts where the correct output is a non-answer.
• Model invocation logging is off by default; if you didn’t turn it on, there is nothing to investigate after an incident.
• Prompt caching helps only when a stable prefix repeats. A prompt that changes at the top every call caches nothing, and the common TTL is five minutes, so an idle cache expires.
• Output tokens usually cost more than input tokens, and on several current models one output token draws 5 to 15 tokens of quota. Trimming a rambling response helps on both counts.
• An average does not show tail latency. Track p50 and p99; a good mean with an ugly p99 still fails real users.
• Latency-optimised inference is a preview feature. Once you reach its quota for a model, requests fall back to standard latency and bill at standard rates.
• Geographic cross-Region inference profiles keep processing inside a geography such as US or EU. Global profiles route to any commercial Region and price around 10% lower, so decide on data residency before cost.
• Throttling is expected under load. Without retries and back-off, throttles surface to users as hard errors.
• Fewer, tighter retrieval chunks cut both cost and latency; stuffing the context window wastes tokens and can dilute the answer.

Say it in one line

1. The golden set is the ruler; it must carry hard and out-of-scope cases, not just happy paths.
2. Built-in metrics scale, a judge model scales with nuance, humans decide the hard calls.
3. Bedrock has evaluation jobs for plain models and for RAG pipelines.
4. Faithfulness is “supported by the retrieved text”; correctness is “right about the world”.
5. Human labels come from a private work team of up to 50, held to a rubric; production review is a loop you assemble.
6. Turn on Bedrock model invocation logging to S3 or CloudWatch before you need it.
7. The agent trace tells you which step failed; CloudWatch metrics tell you how often.
8. You pay per input and output token, and output usually costs more and draws more quota.
9. Provisioned Throughput is hourly, on no-commitment, 1-month or 6-month terms; batch is half price for work that can wait.
10. Prompt caching reuses a stable prefix; response and semantic caching skip the model for repeat prompts.
11. Intelligent Prompt Routing picks between two models in one family per request; smaller models cut both cost and latency.
12. Tame the bill with Budgets, cost allocation tags, application inference profiles, Service Quotas, and rate limiting.
13. Stream, alarm on TimeToFirstToken, and report p50 and p99, never just the average.
14. Ship staged rollouts with versions and aliases, keep retries and throttling in place, and use cross-Region inference profiles for failover.