The managed jobs are worth knowing by their official name because scenarios use it. They cover the dataset-scoring half of evaluation well, and stop short of two things the neighbouring cards handle: scoring an agent’s whole run rather than its last message, and turning a score into a gate that a pipeline can act on.
Flash card
Amazon Bedrock Model Evaluations: managed evaluation jobs that score model outputs over a dataset and write per-example results to Amazon S3, without you building a harness.
- It runs three job shapes, and each answers a different question. Automatic evaluation scores a dataset against built-in metrics and is the cheapest way to compare candidates. An LLM-as-a-Judge job has one model grade another’’s answers on criteria you choose, which reaches qualities no built-in metric captures. A human evaluation job routes examples to a workforce you bring, your own team or a managed one, and is the only shape that reflects a real reader’’s judgement.
- The task type you pick constrains the built-in metrics on offer, so a summarisation job and a classification job score on different things. Choosing the task type badly is how a multi-model evaluation ends up comparing candidates on a metric nobody in the room cares about.
- RAG evaluation in Amazon Bedrock Knowledge Bases reports retrieval quality and response quality as separate numbers rather than one blended score, which is what settles an argument about whether a bad answer came from the retriever pulling the wrong passages or the model mishandling passages that were fine.
- Every job writes its results per example as JSON to an S3 output location, so the durable artefact is a file you can diff between runs, feed into cost-performance analysis alongside token counts and latency, and keep as evidence for an audit long after the console view has moved on.
- {“A judge job is cheaper and covers far more examples than human review, and it carries known biases”=>”position bias, where the ordering of two candidate answers moves the verdict, and self-preference bias, where a model favours text that reads like its own. A stratified human sample over the same examples is what calibrates the judge and tells you how far to trust it.”}
Pick it when
Pick it when a scenario wants many models or many examples scored on a schedule someone else operates: choosing between foundation models before a launch, re-scoring a golden set after a prompt or model version change, or collecting human feedback through an interface you did not have to build. It is the managed answer to “how do we know this one is better”, and it produces comparable numbers across candidates.
It's the wrong answer when
It is the wrong answer for scoring an agent’s trajectory, because it sees the final output and never the tool calls, the reasoning steps, or the order they happened in; that needs trace-level assessment of the run. It is also wrong for continuous production monitoring, since a job is something you launch over a dataset rather than something that watches live traffic. Scheduled replay of a golden set, or a canary exercising the live endpoint, covers that ground.