The managed jobs are worth knowing by their console name, since scenarios use it: Evaluations, under Inference and assessment. They cover the dataset-scoring half of evaluation. They stop short of two things the neighbouring cards handle: scoring an agent’s whole run rather than its last message, and turning a score into a gate a pipeline can act on.
Flash card
Amazon Bedrock evaluations: managed jobs that score model or RAG output over a prompt dataset and write per-record results to Amazon S3, with no harness to build.
- Model evaluation runs three job shapes. An automatic job scores responses on built-in accuracy, robustness and toxicity metrics. A model-as-a-judge job has an evaluator model score a generator model on metrics such as Builtin.Correctness and Builtin.Faithfulness, with an explanation per record. A human job sends records to a private work team you bring, capped at 50 workers; the SageMaker Ground Truth workforce behind it is closed to new customers.
- {“You choose one task type per job; in an automatic job it fixes the metrics and built-in datasets on offer”=>”general text generation, text summarisation, question and answer, or text classification. Summarisation accuracy computes BERTScore; classification accuracy computes a classification accuracy score.”}
- An automatic or judge job evaluates one generator model, so ranking candidates means one job each over the same dataset and metrics. A human job takes up to two inference sources and rates them side by side.
- RAG evaluation scores a retrieve-only job on Builtin.ContextRelevance and Builtin.ContextCoverage, and a retrieve-and-generate job on correctness, faithfulness, citation precision and seven more. Separate numbers settle whether a bad answer came from retrieval or generation.
- Results land in the S3 output location as JSON Lines, one record per prompt with the input, the model responses and the scores, so two runs diff cleanly. A custom prompt dataset holds at most 1,000 prompts.
- An automatic job bills only the model inference. Judge and RAG jobs bill the evaluator model’’s tokens at on-demand standard-tier prices, and a human job adds USD$0.21 per completed human task.
Pick it when
Pick it when a scenario wants many examples scored by a managed job: choosing between foundation models before a launch, re-scoring a golden set after a prompt or model change, or collecting human ratings without building the interface.
It's the wrong answer when
It is wrong for scoring an agent’s run: it reads the final output, not the tool calls or their order. Amazon Bedrock AgentCore Evaluations scores agent traces against an expected tool trajectory, and its online evaluation watches live traffic, which a dataset job never touches.