The situation
A team is shipping a document-processing assistant on AWS. It reads incoming supplier contracts, pulls out key terms, classifies each clause by risk, and drafts a plain-language summary for a reviewer. The model is a Claude model on Amazon Bedrock, fronted by a retrieval layer over the company’s own policy library.
Three separate needs for human judgement have shown up, and the team keeps confusing them. First, the risk classifier was fine-tuned on a few thousand hand-labelled clauses, and they want a larger, cleaner labelled set to improve it, plus some preference data where a person ranks two candidate summaries against each other. Second, in production, when the classifier’s confidence on a clause drops below a line, or the clause touches liability or indemnity, they want a human to check the call before it lands in the reviewer’s queue. Third, before they roll the next model version out, they want people to judge whether its summaries actually read better, which no automated score has settled for them.
All three got written up in one ticket as “add human review”. They are three different jobs, and on AWS they map to three different services. Picking the wrong one for a given need means either building a labelling pipeline where a review gate belonged, or standing up a production review workflow when what they wanted was an offline quality judgement.
What actually matters
The first thing to pin down is which of the three jobs a given need actually is, because they sit at different points in the lifecycle and produce different things. Producing labelled or ranked data feeds training: it happens before or between model versions, its output is a dataset, and its consumer is a fine-tuning or preference-optimisation job. Reviewing a prediction happens in production, inline with a live request, and its output is a corrected or confirmed result for that one item. Evaluating a model happens at a decision point, before or during a rollout, and its output is a quality judgement about the model as a whole, not about any single live request. Data, decisions, verdicts: three outputs, three moments.
The second axis is what kind of judgement is being asked for. A confidence threshold or a high-stakes rule (“route anything touching indemnity to a person”) is a routing decision: the machine decides who looks at the item, and the human gives a verdict on that one item. That is different from a subjective quality judgement (“is this summary clearer, is this tone right, did it miss the point”), which is the sort of thing automated metrics and an LLM acting as a judge approximate but never fully capture. When the judgement is inherently subjective and you need it aggregated across many outputs to compare models, that is evaluation, not per-item review.
The third is the cost and latency of inserting a person. A human in a live request path adds seconds to minutes and a per-item labour cost, and it only pays off when the stakes of an automated error are high enough to justify the wait, or when confidence is genuinely low. Putting a human on every prediction defeats the point of the model; the discipline is routing only the items that need it. Labelling and evaluation, by contrast, are offline, so latency barely matters and the cost is a batch cost you plan for, not a tax on every request.
The fourth is the stakes of an error, which decides how much human coverage is worth buying at each point. Bad training labels quietly poison every future prediction, so label quality is worth real investment even though nobody sees it directly. A wrong live prediction on a liability clause has immediate consequences, which is exactly what a review gate is for. A model that reads worse than its predecessor is a reversible mistake if evaluation catches it before rollout and an expensive one if it doesn’t.
The softer cross-cutting point: all three can draw on the same pool of people. AWS lets you bring your own private workforce, use a third-party vendor, or use a public crowd for the data-labelling jobs, and lets human evaluation use either an AWS-managed team or your own workforce. The workforce is a shared resource; the service you wrap around it is what changes with the job.
What we’ll filter on
- Which job is it, producing training or preference data, reviewing a live prediction, or evaluating a model’s quality?
- Is the trigger a confidence threshold or high-stakes rule, or is it a subjective quality judgement?
- Where in the lifecycle does it sit, offline before or between versions, or inline with a production request?
- Latency and cost tolerance, can it be a planned batch, or does it block a live request?
- Stakes of an error at that point, and therefore how much human coverage is worth buying.
The human-in-the-loop landscape
Amazon SageMaker Ground Truth. A managed data-labelling service for building labelled training datasets. You define a labelling task (classification, bounding boxes, text spans, ranking, and so on), point it at your raw data, and send the work to a workforce: your own private team, an approved third-party vendor, or a public crowd through Amazon Mechanical Turk. It supports the labelling patterns that matter for generative AI, including ranking and preference comparisons where a worker picks the better of two model outputs, which is the raw material for reinforcement learning from human feedback and other preference-tuning approaches. Ground Truth also offers a fully managed option where AWS operates the labelling pipeline for you. The output is a dataset, and its consumer is a training or tuning job. It is not a place to review live production traffic and it is not where you judge a finished model.
Amazon Augmented AI (A2I). A service for routing individual production predictions to human reviewers inside a live workflow. You define a human review workflow with an activation condition, typically a confidence threshold or a business rule, and when an inference meets that condition A2I pulls it out of the automated path, presents it to a reviewer through a customisable UI, and returns the human’s answer so your application can proceed. The reviewers can be a private workforce, a vendor, or a public crowd. A2I is built for the “check the low-confidence or high-stakes call before it counts” job: it operates per-item, inline, on live data. It is not a bulk labelling tool for building a training set, and it is not a model-quality evaluation.
Human evaluation in Amazon Bedrock model evaluation jobs. Bedrock’s model evaluation feature runs jobs that score a model’s outputs, and alongside the automated metric-based jobs it offers human evaluation jobs, where human raters judge output quality against criteria you define: helpfulness, coherence, tone, correctness, relevance, whatever the subjective bar is. You can use an AWS-managed evaluation team or bring your own workforce, and you point the job at prompts and one or more models. The output is an aggregated quality verdict that lets you compare models or gate a rollout on subjective quality that automated metrics and LLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. scoring miss. It sits at a decision point in the lifecycle, offline, judging the model rather than servicing a live request.
To place them on the lifecycle:
Side by side
| SageMaker Ground Truth | Augmented AI (A2I) | Bedrock human evaluation | |
|---|---|---|---|
| Job | Build labelled / ranked data | Review a live prediction | Judge model quality |
| Lifecycle moment | Before / between versions | In production, inline | At a rollout decision point |
| Output | A dataset | A per-item verdict | An aggregated quality verdict |
| Trigger | You choose what to label | Confidence threshold / high-stakes rule | You choose prompts and models |
| Judgement type | Annotation, ranking | Confirm or correct one item | Subjective quality criteria |
| Latency sensitivity | Offline batch | Blocks a live request | Offline batch |
| Preference / RLHF data | ✓ | ✗ | ✗ |
| Per-item production gate | ✗ | ✓ | ✗ |
| Compare models before rollout | ✗ | ✗ | ✓ |
| Workforce options | Private, vendor, crowd; managed option | Private, vendor, crowd | AWS-managed team or your own |
Reading the table against the three needs: the larger labelled set and the preference ranking are Ground Truth; the low-confidence and liability-clause review is A2I; the “does the new version read better” judgement is a Bedrock human evaluation job. One ticket, three services.
The picks in depth
Ground Truth for the training and preference data. The team wants two things here, and Ground Truth covers both. The larger labelled clause set is a straightforward classification labelling task: define the risk categories, feed in the raw clauses, and send the work to a workforce. Because label quality silently determines every future prediction, this is where investing in a trusted private workforce or a vetted vendor, rather than the cheapest crowd, tends to pay off, and where the annotation guidelines matter as much as the tool. The preference data is the ranking pattern: show a worker two candidate summaries for the same contract and have them pick the better one, producing exactly the comparative signal that preference-tuning and RLHF-style training consume. The managed option exists if the team would rather AWS run the labelling operation than staff and operate it themselves. What Ground Truth is not: a place to intercept live traffic. Its output is a file of labels destined for a training job, full stop.
A2I for the production review gate. This is the per-item, inline job. The team defines a human review workflow whose activation condition captures the two cases they care about: confidence below their chosen line, and any clause the classifier tags as liability or indemnity. Items that don’t trip the condition flow straight through untouched, which is the whole point; only the flagged ones pull a reviewer in, so the human cost tracks the genuinely uncertain and genuinely high-stakes fraction rather than every request. The reviewer sees the item in a task UI, gives the corrected or confirmed answer, and the workflow returns it so the pipeline continues. The design tension to hold is where the threshold sits: too low and everything routes to a person and the automation is pointless; too high and risky calls slip through unreviewed. That threshold is a dial you tune against the error stakes, and it is the substance of using A2I well.
Bedrock human evaluation for the rollout decision. The “is the new version actually better” question is subjective and about the model as a whole, so it is neither a labelling task nor a per-item gate. A Bedrock model evaluation job with human raters is the fit: point it at a representative set of contract-summary prompts, define the criteria (clarity, faithfulness to the source clause, tone), and have raters judge, using an AWS-managed evaluation team or the team’s own reviewers. The output aggregates into a comparison that tells them whether to promote the new model. This is the human counterpart to the automated scoring covered when an LLM stands in as the judge: the automated job is cheap and fast and catches regressions on measurable properties, and the human job is slower and dearer and catches the subjective quality the automated scores keep missing. Most teams run both and reserve human evaluation for the criteria that genuinely need a person.
A worked example: routing the three needs
The team walks each need through the filter.
The larger labelled clause set: the job is build training data, the output is a dataset, and it happens between model versions with no latency pressure. That is Ground Truth, classification task, private workforce because label quality feeds every future prediction. The preference ranking is the same service, ranking task, output feeding the preference-tuning job.
The liability-and-indemnity review: the job is review a live prediction, the trigger is a high-stakes rule plus a confidence threshold, it sits inline in production, and it blocks the item until a person answers. That is A2I, with an activation condition combining the confidence line and the clause-type rule, and a threshold tuned so only the uncertain and the high-stakes fraction reaches a reviewer.
The “does v2 read better” question: the job is evaluate model quality, the judgement is subjective, it sits at the rollout decision point, and it is an offline batch. That is a Bedrock human evaluation job over a representative prompt set, run alongside the automated evaluation, with a managed or in-house rater team.
Three needs, three filters, three services, and none of them substitutable for the others. The failure the team started with was letting all three wear the label “human review” and reaching for one tool to do all three jobs.
What’s worth remembering
- Human judgement plugs into a generative-AI system in three distinct jobs: producing training or preference data, reviewing live predictions, and evaluating model quality; they use three different AWS services.
- SageMaker Ground Truth builds labelled and ranked training datasets, including the preference comparisons that feed RLHF-style tuning, using a private workforce, a vendor, a public crowd, or a fully managed option.
- Augmented AI (A2I) routes individual production predictions to human reviewers when a confidence threshold or a high-stakes rule fires; it is per-item, inline, and blocks the live request until a person answers.
- Bedrock model evaluation jobs offer human evaluation, where raters judge subjective output quality that automated metrics miss, using an AWS-managed team or your own workforce, at a rollout decision point.
- The deciding question is the job: producing data (Ground Truth), reviewing a live item (A2I), or judging the model as a whole (Bedrock human evaluation).
- A confidence threshold or high-stakes rule is a routing decision for A2I; a subjective quality judgement aggregated across outputs is evaluation, not per-item review.
- Human review in the live path costs latency and per-item labour, so route only the low-confidence and high-stakes fraction; a person on every prediction defeats the model.
- Labelling and evaluation are offline batches where latency barely matters; the cost is planned, not a tax on every request.
- Bad training labels silently degrade every future prediction, so label quality is worth investing in even though nobody sees it directly.
- All three jobs can draw on the same workforce pool; the service you wrap around the people is what changes with the job.