The situation
A team is shipping a document-processing assistant on AWS. It reads incoming supplier contracts, pulls out key terms, classifies each clause by risk, and drafts a plain-language summary for a reviewer. The model is a Claude model on Amazon Bedrock, fronted by a retrieval layer over the company’s own policy library.
Three separate needs for human judgement have shown up, and the team keeps confusing them. First, the risk classifier was fine-tuned on a few thousand hand-labelled clauses, and they want a larger, cleaner labelled set to improve it, plus some preference data where a person ranks two candidate summaries against each other. Second, in production, when the classifier’s confidence on a clause drops below a line, or the clause touches liability or indemnity, they want a human to check the call before it lands in the reviewer’s queue. Third, before they roll the next model version out, they want people to judge whether its summaries actually read better, which no automated score has settled for them.
All three got written up in one ticket as “add human review”. They are three different jobs with three different shapes, and mistaking one for another means either building a labelling pipeline where a review gate belonged, or standing up a production review workflow when what they wanted was an offline quality judgement.
What actually matters
The first thing to pin down is which of the three jobs a given need actually is, because they sit at different points in the lifecycle and produce different things. Producing labelled or ranked data feeds training: it happens before or between model versions, its output is a dataset, and its consumer is a fine-tuning or preference-optimisation job. Reviewing a prediction happens in production, inline with a live request, and its output is a corrected or confirmed result for that one item. Evaluating a model happens at a decision point, before or during a rollout, and its output is a quality judgement about the model as a whole, not about any single live request. Data, decisions, verdicts: three outputs, three moments.
The second axis is what kind of judgement is being asked for. A confidence threshold or a high-stakes rule (“route anything touching indemnity to a person”) is a routing decision: the machine decides who looks at the item, and the human gives a verdict on that one item. That is different from a subjective quality judgement (“is this summary clearer, is this tone right, did it miss the point”), which is the sort of thing automated metrics and an LLM acting as a judge approximate but never fully capture. When the judgement is inherently subjective and you need it aggregated across many outputs to compare models, that is evaluation, not per-item review.
The third is the cost and latency of inserting a person. A human in a live request path adds seconds to minutes and a per-item labour cost, and it only pays off when the stakes of an automated error are high enough to justify the wait, or when confidence is genuinely low. Putting a human on every prediction defeats the point of the model; route only the items that need it. Labelling and evaluation, by contrast, are offline, so latency barely matters and the cost is a batch cost you plan for, not a tax on every request.
The fourth is the stakes of an error, which decides how much human coverage is worth buying at each point. Bad training labels quietly poison every future prediction, so label quality is worth real investment even though nobody sees it directly. A wrong live prediction on a liability clause has immediate consequences, which is exactly what a review gate is for. A model that reads worse than its predecessor is a reversible mistake if evaluation catches it before rollout and an expensive one if it doesn’t.
One practical point cuts across all three: they can draw on the same pool of people. Whether the people are an in-house team or a partner workforce, the workforce is a shared resource; the workflow you wrap around it is what changes with the job. What changed in June 2026 is how much of that wrapping AWS still sells. SageMaker Ground Truth and Amazon Augmented AI (A2I), the managed services for the labelling and review jobs, moved to maintenance and close to new customers from the end of July; teams already running on them keep running, and everyone else builds the wrapping themselves.
What we’ll filter on
- Which job is it, producing training or preference data, reviewing a live prediction, or evaluating a model’s quality?
- Is the trigger a confidence threshold or high-stakes rule, or is it a subjective quality judgement?
- Where in the lifecycle does it sit, offline before or between versions, or inline with a production request?
- Latency and cost tolerance, can it be a planned batch, or does it block a live request?
- Stakes of an error at that point, and therefore how much human coverage is worth buying.
The landscape
The labelling workflow (SageMaker Ground Truth for teams already on it). A labelling workflow builds labelled training datasets: define a labelling task (classification, bounding boxes, text spans, ranking, and so on), point it at your raw data, and send the work to a workforce. It covers the labelling patterns that matter for generative AI, including ranking and preference comparisons where a worker picks the better of two model outputs, which is the raw material for reinforcement learning from human feedback and other preference-tuning approaches. Amazon SageMaker Ground Truth is the managed service that ran this job, with a private team, an approved vendor, or a public crowd through Amazon Mechanical Turk as the workforce; both services moved to maintenance in June 2026 and close to new customers from the end of July, and the fully managed variant, Ground Truth Plus, reached end of support in June. Existing labelling workflows keep running. A fresh build brings its own annotators or a partner workforce with its own tooling; there is no like-for-like managed replacement. Whoever runs it, the output is a dataset, and its consumer is a training or tuning job. It is not a place to review live production traffic and it is not where you judge a finished model.
The review gate (Amazon Augmented AI for teams already on it). A review gate routes individual production predictions to human reviewers inside a live workflow. You define an activation condition, typically a confidence threshold or a business rule, and when an inference meets that condition the gate pulls it out of the automated path, presents it to a reviewer, and returns the human’s answer so your application can proceed. Amazon Augmented AI (A2I) is the managed service that shipped this pattern ready-made, with a customisable review UI and workforce options; it moved to maintenance in June 2026 alongside Ground Truth, closes to new customers from the end of July, and keeps running for the workflows already on it. A fresh build assembles the same gate from primitives: hold the flagged item in a Step Functions workflow or an SQS queue, present it in a reviewer UI you own, and feed the verdict back into the pipeline. Either way the gate is built for the “check the low-confidence or high-stakes call before it counts” job: it operates per-item, inline, on live data. It is not a bulk labelling tool for building a training set, and it is not a model-quality evaluation.
Human evaluation in Amazon Bedrock model evaluation jobs. Bedrock’s model evaluation feature runs jobs that score a model’s outputs, and alongside the automated metric-based jobs it offers human evaluation jobs, where human raters judge output quality against criteria you define: helpfulness, coherence, tone, correctness, relevance, whatever the subjective bar is. You bring the workforce, an internal team or a vendor, and you point the job at prompts and one or more models. The output is an aggregated quality verdict that lets you compare models or gate a rollout on subjective quality that automated metrics and LLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. scoring miss. It sits at a decision point in the lifecycle, offline, judging the model rather than servicing a live request.
To place them on the lifecycle:
Evaluation
Side by side
| Labelling workflow | Review gate | Bedrock human evaluation | |
|---|---|---|---|
| Job | Build labelled / ranked data | Review a live prediction | Judge model quality |
| Lifecycle moment | Before / between versions | In production, inline | At a rollout decision point |
| Output | A dataset | A per-item verdict | An aggregated quality verdict |
| Trigger | You choose what to label | Confidence threshold / high-stakes rule | You choose prompts and models |
| Judgement type | Annotation, ranking | Confirm or correct one item | Subjective quality criteria |
| Latency sensitivity | Offline batch | Blocks a live request | Offline batch |
| Preference / RLHF data | ✓ | ✗ | ✗ |
| Per-item production gate | ✗ | ✓ | ✗ |
| Compare models before rollout | ✗ | ✗ | ✓ |
| Workforce options | Own team or partner workforce | Own team or partner workforce | Workforce you bring |
Reading the table against the three needs: the larger labelled set and the preference ranking are the labelling workflow; the low-confidence and liability-clause review is the review gate; the “does the new version read better” judgement is a Bedrock human evaluation job. One ticket, three workflows. Teams already running Ground Truth and A2I get the first two ready-made; a new build assembles them and gets the same three-way split.
The solution
A labelling workflow for the training and preference data. The team wants two things here, and the same workflow covers both. The larger labelled clause set is a straightforward classification labelling task: define the risk categories, feed in the raw clauses, and send the work to a workforce. Because label quality silently determines every future prediction, this is where investing in a trusted in-house team or a vetted partner workforce tends to pay off, and where the annotation guidelines matter as much as the tool. The preference data is the ranking pattern: show a worker two candidate summaries for the same contract and have them pick the better one, producing exactly the comparative signal that preference-tuning and RLHF-style training consume. SageMaker Ground Truth ships this workflow ready-made for the teams already on it; since the service moved to maintenance in June 2026 and closes to new customers from the end of July, a team starting now runs its own annotators or a partner workforce with its own tooling, holding to the same rubric and the same guidelines. What the labelling workflow is not: a place to intercept live traffic. Its output is a file of labels destined for a training job, full stop.
A review gate for the production predictions. This is the per-item, inline job. The team defines a review workflow whose activation condition captures the two cases they care about: confidence below their chosen line, and any clause the classifier tags as liability or indemnity. Items that don’t trip the condition flow straight through untouched; only the flagged ones pull a reviewer in, so the human cost tracks the genuinely uncertain and genuinely high-stakes fraction rather than every request. The reviewer sees the item in a task UI, gives the corrected or confirmed answer, and the workflow returns it so the pipeline continues. A2I packaged exactly this and still runs it for existing customers; a new build assembles the gate from primitives, a Step Functions workflow or an SQS queue holding the flagged item, a reviewer UI the team owns, and a callback that resumes the pipeline with the verdict. The design tension is the same either way: where the threshold sits. Too low and everything routes to a person and the automation is pointless; too high and risky calls slip through unreviewed. That threshold is a dial you tune against the error stakes, and it is the substance of the gate.
Bedrock human evaluation for the rollout decision. The “is the new version actually better” question is subjective and about the model as a whole, so it is neither a labelling task nor a per-item gate. A Bedrock model evaluation job with human raters is the fit: point it at a representative set of contract-summary prompts, define the criteria (clarity, faithfulness to the source clause, tone), and have raters judge, using a workforce the team brings. The output aggregates into a comparison that tells them whether to promote the new model. This is the human counterpart to the automated scoring covered when an LLM stands in as the judge: the automated job is cheap and fast and catches regressions on measurable properties, and the human job is slower and dearer and catches the subjective quality the automated scores keep missing. Most teams run both and reserve human evaluation for the criteria that genuinely need a person.
Worked example
The team walks each need through the filter.
The larger labelled clause set: the job is build training data, the output is a dataset, and it happens between model versions with no latency pressure. That is the labelling workflow, a classification task run by a trusted workforce because label quality feeds every future prediction. The preference ranking is the same workflow, ranking task, output feeding the preference-tuning job.
The liability-and-indemnity review: the job is review a live prediction, the trigger is a high-stakes rule plus a confidence threshold, it sits inline in production, and it blocks the item until a person answers. That is the review gate, with an activation condition combining the confidence line and the clause-type rule, and a threshold tuned so only the uncertain and the high-stakes fraction reaches a reviewer.
The “does v2 read better” question: the job is evaluate model quality, the judgement is subjective, it sits at the rollout decision point, and it is an offline batch. That is a Bedrock human evaluation job over a representative prompt set, run alongside the automated evaluation, with an in-house or vendor rater team.
Three needs, three filters, three workflows, and none of them substitutable for the others. The failure the team started with was letting all three carry the label “human review” and reaching for one tool to do all three jobs.
What’s worth remembering
- Human judgement plugs into a generative-AI system in three distinct jobs: producing training or preference data, reviewing live predictions, and evaluating model quality; each needs its own workflow.
- A labelling workflow builds labelled and ranked training datasets, including the preference comparisons that feed RLHF-style tuning; SageMaker Ground Truth ran this as a managed service but moved to maintenance in June 2026 and closes to new customers from the end of July, so a fresh build brings its own annotators or a partner workforce with its own tooling.
- A review gate routes individual production predictions to human reviewers when a confidence threshold or a high-stakes rule fires; it is per-item, inline, and blocks the live request until a person answers. A2I ships it ready-made for existing customers only; a new build assembles it from Step Functions or SQS plus a reviewer UI it owns.
- Bedrock model evaluation jobs offer human evaluation, where raters judge subjective output quality that automated metrics miss, with a workforce you bring, at a rollout decision point.
- Human review in the live path costs latency and per-item labour, so route only the low-confidence and high-stakes fraction; a person on every prediction defeats the model.