Where Humans Belong in a GenAI Pipeline

The situation

A team is shipping a document-processing assistant on AWS. It reads incoming supplier contracts, pulls out key terms, classifies each clause by risk, and drafts a plain-language summary for a reviewer. The model is a Claude model on Amazon Bedrock, fronted by a retrieval layer over the company’s own policy library.

Three separate needs for human judgement have shown up, and the team keeps confusing them. First, the risk classifier was fine-tuned on a few thousand hand-labelled clauses, and they want a larger, cleaner labelled set to improve it, plus some preference data where a person ranks two candidate summaries against each other. Second, in production, when the classifier’s confidence on a clause drops below a line, or the clause touches liability or indemnity, they want a human to check the call before it lands in the reviewer’s queue. Third, before they roll the next model version out, they want people to judge whether its summaries actually read better, which no automated score has settled for them.

All three got written up in one ticket as “add human review”. They are three different jobs with three different shapes, and mistaking one for another means either building a labelling pipeline where a review gate belonged, or standing up a production review workflow when what they wanted was an offline quality judgement.

What actually matters

Start by pinning down which of the three jobs a need actually is. They sit at different points in the lifecycle and produce different things. Producing labelled or ranked data feeds training. It happens before or between model versions, its output is a dataset, and a fine-tuning or preference-optimisation job consumes it. Reviewing a prediction happens in production, inline with a live request, and returns a corrected or confirmed result for that one item. Evaluating a model happens at a decision point, before or during a rollout, and returns a quality judgement about the model as a whole. Three outputs, three moments.

The second axis is what kind of judgement is being asked for. A confidence threshold or a high-stakes rule (“route anything touching indemnity to a person”) is a routing decision: the rule selects who looks at the item, and the person gives a verdict on that one item. A subjective quality judgement is a different thing. “Is this summary clearer, is the tone right, did it drop the clause that mattered” is what automated metrics and an LLM acting as a judge approximate without fully capturing. When the judgement is subjective and you need it aggregated across many outputs to compare models, that is evaluation, not per-item review.

The third is the latency and labour of inserting a person. A human in a live request path adds seconds to minutes and a per-item labour cost, and that is worth the wait only when an automated error does real damage, or when confidence is genuinely low. Routing every prediction to a person removes the throughput the model was there for. Route only the items that need it. Labelling and evaluation are offline, so latency barely matters and the labour is a planned batch rather than an addition to every request.

The fourth is the stakes of an error, which decide how much human coverage each point warrants. Bad training labels degrade every future prediction, and nobody sees them directly, so label quality deserves real effort. A wrong live prediction on a liability clause has immediate consequences, which is what a review gate is for. A model that reads worse than its predecessor is a reversible mistake if evaluation catches it before rollout and an expensive one if it doesn’t.

All three can draw on the same pool of people. Whether they are an in-house team or a partner workforce, the workforce is a shared resource, and the workflow around it is what changes with the job. What changed in mid-2026 is how much of that workflow AWS still supplies. SageMaker Ground Truth and Amazon Augmented AI (A2I), the managed services for the labelling and review jobs, closed to new customers on 30 July 2026 and are now in maintenance. Existing customers keep running and AWS has said it will add no new features; everyone else builds the workflow themselves.

What we’ll filter on

1. Which job is it, producing training or preference data, reviewing a live prediction, or evaluating a model’s quality?
2. Is the trigger a confidence threshold or high-stakes rule, or is it a subjective quality judgement?
3. Where in the lifecycle does it sit, offline before or between versions, or inline with a production request?
4. Latency and cost tolerance, can it be a planned batch, or does it block a live request?
5. Stakes of an error at that point, and therefore how much human coverage it warrants.

The landscape

The labelling workflow (SageMaker Ground Truth for teams already on it). A labelling workflow builds labelled training datasets: define a labelling task, point it at your raw data, and send the work to a workforce. Amazon SageMaker Ground Truth is the managed service that ran this job. Its built-in task types cover images, video frames, 3D point clouds, named entity recognition, and single- or multi-label text classification. Ranking two model outputs against each other, which is the raw material for reinforcement learning from human feedback and other preference tuning, is not one of them: that is a custom labelling workflow with a worker task template you write. Ground Truth closed to new customers on 30 July 2026, the fully managed variant, Ground Truth Plus, reached end of support on 30 June 2026, and the Amazon Mechanical Turk workforce closes permanently on 30 September 2026, which leaves a private team or an approved vendor. Existing labelling workflows keep running, and AWS has not named a successor service, so a fresh build brings its own annotators or a partner workforce with its own tooling. Whoever runs it, the output is a dataset that a training or tuning job consumes. It is not a place to review live production traffic and it is not where you judge a finished model.

The review gate (Amazon Augmented AI for teams already on it). A review gate routes individual production predictions to human reviewers inside a live workflow. You define an activation condition, typically a confidence threshold or a business rule, and an inference that meets it is pulled out of the automated path, presented to a reviewer, and returned with the reviewer’s answer so your application can proceed. Amazon Augmented AI (A2I) is the managed service that shipped this pattern ready-made, with a customisable review UI and workforce options. It closed to new customers on 30 July 2026 alongside Ground Truth and keeps running for the workflows already on it. A fresh build assembles the same gate from primitives: hold the flagged item in a Step Functions workflow or an SQS queue, present it in a reviewer UI you own, and feed the verdict back into the pipeline. Either way the gate does one job, checking the low-confidence or high-stakes call before it counts, per item, inline, on live data. It is not a bulk labelling tool for building a training set, and it is not a model-quality evaluation.

Step Functions supplies the holding pattern so you don’t write one. A Standard-workflow task state using the .waitForTaskToken pattern issues a task token when a flagged item reaches it and holds that execution open. The reviewer UI receives the item along with the token, and calling SendTaskSuccess with the token resumes the execution with the verdict attached; SendTaskFailure covers the reviewer who rejects the item outright. Two settings cover the reviewer who disappears. HeartbeatSeconds fails the task if no SendTaskHeartbeat arrives inside the interval, catching in minutes the reviewer who opened an item and walked away. TimeoutSeconds sends an abandoned review into a fallback branch, escalation to a second reviewer, or a safe default. Without either, a waiting task sits until the execution hits the one-year service quota, with a contract sitting unsummarised behind it.

The collection side is smaller than it looks, and it is shared. An API Gateway endpoint behind the reviewer UI and one behind the thumbs-up control an end user sees on a finished summary are the same mechanism: a small API that writes a verdict keyed by interaction id, with the model version, the prompt, and the output it judged. Build it once for both callers, and the reviewer’s correction and the end user’s signal land in the same store, readable together later, whether to build the next labelled set or to watch quality drift between releases.

Labelling, the review gate, and human evaluation are the three human-augmentation patterns here. Together they leave the model handling volume, people handling the judgements automation cannot make reliably, and a defined path carrying each verdict back to where it changes something.

Human evaluation in Amazon Bedrock evaluations. Bedrock evaluations runs jobs that score a model’s outputs. Alongside the programmatic metric-based jobs and the judge-model jobs, it offers model evaluation jobs that use human workers: raters score output against metrics you define, each with a rating method (thumbs up/down, choice buttons, an individual or comparison Likert scale, ordinal ranking). You supply the work team, up to 50 workers drawn from a private workforce, and you point the job at a custom prompt dataset of at most 1,000 prompts and at most two inference sources, so a single job compares two models. Creating one through the API needs a SageMaker flow definition ARN, the same A2I construct the review gate uses. The output aggregates into a quality verdict for comparing those two models or gating a rollout on subjective quality that automated metrics and LLM-as-a-judgeLLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. scoring do not measure. It sits at a decision point in the lifecycle, offline, judging the model rather than servicing a live request.

To place them on the lifecycle:

Evaluation

Side by side

	Labelling workflow	Review gate	Bedrock human evaluation job
Job	Build labelled / ranked data	Review a live prediction	Judge model quality
Lifecycle moment	Before / between versions	In production, inline	At a rollout decision point
Output	A dataset	A per-item verdict	An aggregated quality verdict
Trigger	You choose what to label	Confidence threshold / high-stakes rule	You choose prompts and models
Judgement type	Annotation, ranking	Confirm or correct one item	Subjective quality metrics
Latency sensitivity	Offline batch	Blocks a live request	Offline batch
Preference / RLHF data	✓	✗	✗
Per-item production gate	✗	✓	✗
Compare models before rollout	✗	✗	✓ (two at a time)
Workforce	Own team or vendor	Own team or vendor	Private work team, up to 50

Reading the table against the three needs: the larger labelled set and the preference ranking are the labelling workflow; the low-confidence and liability-clause review is the review gate; the “does the new version read better” judgement is a Bedrock human evaluation job. One ticket, three workflows. Teams already on Ground Truth and A2I have the first two ready-made. A new build assembles them and lands on the same three-way split.

The solution

A labelling workflow for the training and preference data. The team wants two things here, and the same workflow covers both. The larger labelled clause set is a straightforward classification labelling task: define the risk categories, feed in the raw clauses, and send the work to a workforce. Label quality determines every future prediction without ever being visible, so a trusted in-house team or a vetted partner workforce repays the effort of assembling it, and the annotation guidelines matter as much as the tool. The preference data is the ranking pattern: show a worker two candidate summaries for the same contract and have them pick the better one, producing the comparative signal that preference tuning and RLHF-style training consume. Ground Truth ran the classification half as a built-in task type and the ranking half as a custom template, and it closed to new customers on 30 July 2026, so a team starting now runs its own annotators or a partner workforce with its own tooling, holding to the same rubric and the same guidelines. What the labelling workflow is not: a place to intercept live traffic. Its output is a file of labels destined for a training job, full stop.

A review gate for the production predictions. This is the per-item, inline job. The team defines a review workflow whose activation condition captures the two cases they care about: confidence below their chosen line, and any clause the classifier tags as liability or indemnity. Items that don’t trip the condition flow straight through untouched. Only the flagged ones reach a reviewer, so the labour tracks the genuinely uncertain and genuinely high-stakes fraction rather than every request. The reviewer sees the item in a task UI, gives the corrected or confirmed answer, and the workflow returns it so the pipeline continues. A2I packaged exactly this and still runs it for existing customers. A new build assembles the gate from primitives: a Step Functions workflow or an SQS queue holding the flagged item, a reviewer UI the team owns, and a callback that resumes the pipeline with the verdict. The design tension is the same either way, and it is where the threshold sits. Too low and everything routes to a person and the automation is pointless; too high and risky calls slip through unreviewed. That threshold is a dial tuned against the error stakes.

A Bedrock human evaluation job for the rollout decision. The “is the new version actually better” question is subjective and about the model as a whole, so it is neither a labelling task nor a per-item gate. A model evaluation job with human workers fits: point it at a representative set of contract-summary prompts, define the metrics (clarity, faithfulness to the source clause, tone) and a rating method for each, and have the team’s own raters score them. With two inference sources allowed per job, the current model and the candidate go head to head, and the output aggregates into a comparison that says whether to promote the new one. This is the human counterpart to the automated scoring covered when an LLM stands in as the judge. The automated job needs no rater time and catches regressions on measurable properties; the human job needs a work team and catches the subjective quality automated scores do not measure. Most teams run both and reserve human evaluation for the metrics that genuinely need a person.

Worked example

The team walks each need through the filter.

The larger labelled clause set: the job is build training data, the output is a dataset, and it happens between model versions with no latency pressure. That is the labelling workflow, a classification task run by a trusted workforce because label quality feeds every future prediction. The preference ranking is the same workflow with a ranking template, its output feeding the preference-tuning job.

The liability-and-indemnity review: the job is review a live prediction, the trigger is a high-stakes rule plus a confidence threshold, it sits inline in production, and it blocks the item until a person answers. That is the review gate. Its activation condition combines the confidence line and the clause-type rule, with a threshold tuned so only the uncertain and high-stakes fraction reaches a reviewer.

The “does v2 read better” question: the job is evaluate model quality, the judgement is subjective, it sits at the rollout decision point, and it runs offline in a batch. That is a Bedrock human evaluation job over a representative prompt set, the two model versions as the two inference sources, run alongside the automated evaluation.

Three needs, three filters, three workflows, and none of them substitutable for the others. The failure the team started with was letting all three carry the label “human review” and reaching for one tool to do all three jobs.

What’s worth remembering

1. Human judgement plugs into a generative-AI system in three distinct jobs: producing training or preference data, reviewing live predictions, and evaluating model quality; each needs its own workflow.
2. A labelling workflow builds labelled and ranked training datasets, including the preference comparisons that feed RLHF-style tuning; SageMaker Ground Truth ran this as a managed service but closed to new customers on 30 July 2026, so a fresh build brings its own annotators or a partner workforce with its own tooling.
3. Ground Truth’s built-in task types stop at classification, entity, image, video and point-cloud work; ranking two model outputs against each other is a custom labelling workflow with a template you write.
4. A review gate routes individual production predictions to human reviewers when a confidence threshold or a high-stakes rule fires, per item, inline, blocking the live request until a person answers. A2I ships it ready-made for existing customers only; a new build assembles it from a Step Functions .waitForTaskToken task or an SQS queue plus a reviewer UI it owns, with HeartbeatSeconds and TimeoutSeconds set so an abandoned review fails instead of parking the execution.
5. A Bedrock model evaluation job with human workers scores the subjective quality automated metrics do not measure, using a private work team of up to 50 raters, at most 1,000 prompts, and at most two inference sources, so it compares two models at a time.
6. Human review in the live path adds latency and per-item labour, so route only the low-confidence and high-stakes fraction; a person on every prediction removes the throughput the model was there for.