The managed jobs are worth knowing by their console name, since scenarios use it: Evaluations, under Inference and assessment. They cover the dataset-scoring half of evaluation. They stop short of two things the neighbouring cards handle: scoring an agent’s whole run rather than its last message, and turning a score into a gate a pipeline can act on.
Flash card
Amazon Bedrock evaluations: managed jobs that score model or RAG output over a prompt dataset and write per-record results to Amazon S3, with no harness to build.
- Model evaluation runs three job shapes. An automatic job scores responses on built-in accuracy, robustness and toxicity metrics. A model-as-a-judge job has an evaluator model score a generator model on metrics such as Builtin.Correctness and Builtin.Faithfulness, with an explanation per record. A human job sends records to a private work team you bring, enrolled through SageMaker Ground Truth and capped at 50 workers.
- {“You choose one task type per job, and it fixes which metrics and built-in datasets are on offer”=>”general text generation, text summarisation, question and answer, or text classification. Summarisation accuracy computes BERTScore; classification accuracy computes a classification accuracy score.”}
- An automatic or judge job evaluates one generator model, so ranking candidates means one job each over the same dataset and metrics. A human job takes up to two inference sources and rates them side by side.
- RAG evaluation scores a retrieve-only job on Builtin.ContextRelevance and Builtin.ContextCoverage, and a retrieve-and-generate job on correctness, faithfulness, citation precision and seven more. Separate numbers settle whether a bad answer came from the retriever or the generation step.
- Results land in the S3 output location as JSON Lines, one record per prompt with the input, the model responses and the scores, so two runs diff against each other. A custom prompt dataset holds at most 1,000 prompts.
- An automatic job bills only the model inference behind it. Judge and RAG jobs bill the evaluator model’’s tokens at on-demand rates, and a human job adds USD$0.21 per completed human task.
Pick it when
Pick it when a scenario wants many examples scored by a managed job: choosing between foundation models before a launch, re-scoring a golden set after a prompt or model version change, or collecting human ratings without building the interface.
It's the wrong answer when
It is wrong for scoring an agent’s run, because it reads the final output and not the tool calls or their order. Amazon Bedrock AgentCore Evaluations scores agent traces against an expected tool trajectory, and its online evaluation watches live traffic, which a dataset job never touches.