Building a Feedback Loop From Users to Model Improvement

The situation

A product team has shipped an AI reply drafter on Amazon Bedrock. It reads a customer support thread, pulls a few relevant help-centre articles through retrieval, and drafts a response the agent can edit and send. It has been live for two months, thousands of drafts a day, and the team has a thumbs up and thumbs down button under each draft that nobody quite trusts.

The numbers look fine and feel wrong. The thumbs-down rate is low, but agents keep rewriting drafts before sending, and a chunk of drafts get discarded entirely. Support leads have a folder of screenshots of bad answers, but nothing connects a screenshot back to the exact prompt, the retrieved articles, and the model version that produced it. When someone asks whether last month’s prompt tweak actually helped, the honest answer is that nobody can measure it.

The team wants to turn all of this into something better than anecdote: capture what users are really telling them, find where the feature fails, and feed that back into changes that are validated before they ship. And they have just noticed that the drafts, the threads, and the corrections are full of customer names, order numbers, and the odd card fragment, so wherever this data goes, it has to be governed.

What actually matters

The instinct is to add more buttons. The thing that actually matters is what happens to a signal after it is collected, because a reaction that lands nowhere is worse than no reaction, it costs attention and returns nothing.

Start with the signal itself. Explicit feedback, the thumbs up or down and any correction the agent types, is high-value and low-volume; people rate a fraction of interactions and they rate the extremes. Implicit feedback is the opposite, abundant and noisy: an agent editing the draft heavily, retrying with a reworded request, or abandoning the draft and writing from scratch all carry information about quality, and they are emitted on nearly every interaction without anyone opting in. A loop that leans only on the explicit signal is reading a biased sample; the implicit signals are what give you coverage.

A signal is only useful if you can trace it back to what produced it. A thumbs-down with no context is a number; a thumbs-down joined to the exact prompt, the retrieved chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. , the model and version, and the final response is a debuggable case. That join is the load-bearing decision in the whole system, and it splits cleanly into two stores: the model invocation log that Bedrock can capture for you, holding the request and response, and your own event store holding the user reaction keyed by the same interaction id. Neither half is useful without the other.

Once cases accumulate, the value is in the clusters, not the individual gripe. A single bad draft is noise; forty bad drafts that all involve refund policy, or all cite the same stale article, or all fail on threads in a particular language, are a diagnosis. Clustering the failures is what turns a screenshot folder into a prioritised list, and each cluster points at a different lever.

The levers matter because most feedback does not call for touching the model at all. A cluster caused by a stale or missing document is a retrieval fix; a cluster caused by wrong tone or format is a prompt or few-shot change; a cluster caused by a consistently unsafe answer is a guardrail rule. Fine-tuning or preference tuning is the heaviest lever and the last one to reach for, justified only when you have enough high-quality, well-labelled preference data that cheaper changes cannot address. Reaching for a training job when a prompt edit would do is the classic over-correction.

And nothing ships on vibes. Every change, prompt, retrieval, or model, gets validated against a held-out evaluation set built from real failures before it reaches users, because a change that fixes one cluster routinely regresses another. The loop is only safe if the validation gate is real, and if you watch for feedback bias, since the users who rate are not the users who do not, and optimising to the raters can quietly degrade the rest.

What we’ll filter on

1. Signal coverage, do you capture both the explicit reactions and the implicit edit, retry, and abandonment signals?
2. Traceability, can every signal be joined back to its exact prompt, retrieved context, response, and model version?
3. Diagnosis, does the design surface failure clusters rather than isolated complaints?
4. Lever fit, does the feedback route to the cheapest change that fixes it, prompt or retrieval before fine-tuning?
5. Validation, is every change measured against a held-out eval set before rollout?
6. Governance, is feedback data treated as potentially containing PII and governed accordingly?

The feedback landscape

Explicit feedback capture. The thumbs up or down, a star rating, or a free-text correction the agent submits alongside the draft. Cheap to add, unambiguous in intent, and directly attributable to one interaction. The limits are volume and bias: only a slice of interactions get rated, and raters skew toward the strongly good and strongly bad. Corrections are the richest form here, because the edited text is close to a gold answer for that case.

Implicit feedback capture. Behavioural signals emitted without the user deciding to give feedback: how much the agent edits the draft before sending (edit distance), whether they retried with a reworded prompt, how long they dwelled, whether they abandoned the draft entirely. Abundant and unbiased by opt-in, but noisy and correlational, a heavy edit might mean a bad draft or a picky agent. Best used in aggregate and as a coverage layer over the sparse explicit signal.

Model invocation logging. Amazon Bedrock can log the full request and response for every invocation to Amazon S3, Amazon CloudWatch Logs, or both, including the prompt, the completion, and metadata like the model id. This is the half of the trace that captures what the model saw and said. Turn it on at the account or region level and you stop reconstructing invocations from memory.

Your own event store. A table or log you control, keyed by interaction id, holding the user reaction, the retrieved chunk ids, the app-side context, and the outcome. Amazon DynamoDB or an S3-based event log both work. This is where the explicit and implicit signals live, and where they join to the invocation log to make a complete, queryable case.

Failure clustering and analysis. Grouping the joined cases to find where the feature fails: by topic, by cited document, by language, by outcome. This can be as simple as querying the event store with Amazon Athena, or as involved as embedding the failed inputs and clustering them. The output is a ranked list of failure modes, each feeding either the eval set or a guardrail rule.

The evaluation set. A curated, held-out collection of real inputs with known-good outputs, drawn straight from the failure clusters. Amazon Bedrock evaluation jobs can score model outputs automatically or through human review against this set. This is the gate: a change is only an improvement if it moves the score without regressing the rest.

Guardrails. Amazon Bedrock Guardrails enforce rules independent of the prompt: denied topicsDenied topicsSubjects you describe in plain language that a Bedrock Guardrail refuses to discuss, whichever way a user phrases the request. , content filters, word filters, contextual grounding checksContextual grounding checkA Guardrail check that tests an answer against the documents it was given and flags claims the source doesn’t support. , and sensitive-information filters that detect or redact PII. Feedback that surfaces a consistent unsafe or off-limits answer becomes a guardrail rule, which is a faster and more reliable fix than hoping a prompt edit holds.

Human labelling. Amazon SageMaker Ground Truth runs human labelling and review workflows, including ranking and preference comparisons. When feedback is going to feed fine-tuning or preference tuning, this is how raw reactions become clean, labelled training data rather than noisy signals.

Fine-tuning and preference tuning. The heavy levers. Fine-tuning customises a model on labelled input-output pairs; preference tuning (DPO or RLHF-style) trains on pairs of preferred and rejected responses, which is exactly the shape a thumbs up or down and a correction produce. On AWS this runs as Bedrock model customisation where the chosen model supports it, or as a training job on SageMaker, and it needs volume and label quality that most clusters will not justify.

Side by side

Option	Captures signal	Traces to context	Finds clusters	Feeds a change	Governs PII	When it fits
Explicit feedback	✓	✗	✗	✓	✗	High-value, low-volume ratings and corrections
Implicit feedback	✓	✗	✗	✓	✗	Broad coverage over the sparse explicit signal
Model invocation logging	✗	✓	✗	✗	needs care	Recording what the model saw and said
Own event store	✓	✓	✗	✗	needs care	Joining reactions to invocations by id
Failure clustering	✗	✗	✓	✓	✗	Turning cases into a ranked list of failure modes
Evaluation set	✗	✗	✗	✓	✗	The gate every change passes before rollout
Guardrails	✗	✗	✗	✓	✓	Enforcing safety rules independent of the prompt
SageMaker Ground Truth	✗	✗	✗	✓	needs care	Clean human labels for training data
Fine-tuning / preference tuning	✗	✗	✗	✓	needs care	Enough labelled preference data to justify it

No single row is the answer. The loop is capture (explicit and implicit) joined through logging and the event store, analysed into clusters, routed to the cheapest fitting lever, and validated against the eval set, with governance sitting across every store that touches user text.

The picks in depth

Capture both signals, and join them by id. Keep the thumbs up or down and the agent’s correction, they are the highest-value data you have, and the correction is nearly a gold answer for that case. But do not stop there, because ratings are sparse and skewed. Record the implicit signals too: edit distance between the draft and what was sent, whether the agent retried, whether the draft was abandoned. The single decision that makes any of it usable is a shared interaction id. Turn on Bedrock model invocation logging so the prompt, retrieved context, and response land in S3 or CloudWatch Logs, write the user reaction to your own event store keyed by that same id, and now every signal joins back to exactly what produced it. Without the join you have two piles of numbers; with it you have cases.

Cluster before you fix. Do not act on individual complaints. Query the joined data, in Athena over the S3 event log, or by embedding failed inputs and grouping them, to find where failures concentrate: a policy topic, a specific stale article, a language, an outcome. Each cluster is a diagnosis, and crucially each points at a different lever. A cluster that all cites one outdated document is not a model problem. Sorting complaints into clusters is what stops the team from fine-tuning away a problem that a document update would have fixed.

Route to the cheapest lever that fits. Most clusters resolve without touching the model. Wrong tone or missing format is a prompt or few-shot change. Stale or absent context is a retrieval fix, reindex the document, adjust chunking, tune what gets fetched. A consistently unsafe or off-limits answer becomes an Amazon Bedrock Guardrails rule, enforced independently of the prompt so it holds regardless of wording. Reach for fine-tuning or preference tuning only when you have accumulated enough high-quality, labelled preference data, the preferred-versus-rejected pairs that a thumbs up or down and a correction naturally form, that cheaper changes genuinely cannot close the gap. That is where Amazon SageMaker Ground Truth earns its place, producing clean human labels and preference rankings, and where a Bedrock model-customisation or SageMaker training job runs. It is the last lever, not the first.

Validate every change against a held-out eval set. Build the evaluation set from the real failure clusters, inputs paired with known-good outputs, and hold it out. Before any change reaches users, score it with an Amazon Bedrock evaluation job, automatic scoring for scale and human review for the subtle cases, and compare against the current version. The point of the gate is regressions: a prompt edit that fixes refund-policy drafts routinely breaks something else, and only a held-out set catches that. Watch feedback bias while you are at it, the agents who rate are not a random sample, so track quality on the whole population, not just on the interactions that got a thumbs down, or you will optimise for the loud minority.

Govern the data end to end. Feedback text is some of the most sensitive data you hold, because it is verbatim user and customer content: names, order numbers, occasionally a card fragment. Treat every store, the invocation logs, the event store, and any training set, as containing PII. Bedrock Guardrails sensitive-information filters can detect and redact PII in flight; lock down access to the logs and event store with least-privilege IAM; set retention so raw feedback does not accumulate forever; and never hand un-redacted user text to a labelling workflow or a training job. Governance is not a final step, it sits across the whole loop.

A worked example: the refund-policy cluster

The team turns on the loop for a fortnight. Invocation logging is on, the thumbs and edit-distance signals write to a DynamoDB table keyed by interaction id, and an Athena query over the joined data ranks the failure modes.

The top cluster is unmistakable: drafts about refund eligibility get thumbs-down at four times the baseline rate, and even the ones sent are heavily edited. Pulling ten cases with their retrieved context shows the cause at once, every draft cites a help-centre article describing last year’s 14-day window; the policy changed to 30 days in the spring, but the old article is still the top retrieval hit. This is not a model failure. The draft faithfully summarised a stale document.

The fix is a retrieval fix: update the article, reindex, and confirm the new version is what gets fetched. Before rolling out, the corrected inputs go into the eval set with known-good 30-day answers, and a Bedrock evaluation job scores the change. Refund-policy accuracy jumps and nothing else regresses, so it ships. A prompt rewrite would have papered over the symptom; fine-tuning would have burned weeks to memorise a fact that belonged in the index. The loop pointed at the right lever because the signal was joined to the context that produced it.

A second, smaller cluster is different in kind: a handful of drafts promised goodwill credit the company does not offer, invented under pressure from an angry thread. Wrong tone is a prompt or few-shot job, but promising a thing that does not exist is a safety rule, so it becomes a Guardrails denied-topic entry that blocks the promise regardless of how the prompt is worded. Two clusters, two levers, one gate they both pass through.

What’s worth remembering

1. A feedback signal only has value if it lands somewhere that changes behaviour; a button that collects reactions into nothing costs attention and returns nothing.
2. Capture both explicit feedback (ratings, corrections) and implicit feedback (edits, retries, abandonment); the explicit signal is high-value but sparse and biased, the implicit signal gives you coverage.
3. The load-bearing decision is a shared interaction id that joins the user reaction in your event store to the prompt, retrieved context, and response in Bedrock model invocation logging.
4. Act on failure clusters, not individual complaints; grouping cases by topic, document, or language turns a screenshot folder into a ranked, diagnosable list.
5. Route each cluster to the cheapest lever that fits: prompt or few-shot for tone and format, retrieval for stale or missing context, a guardrail for safety, and only then fine-tuning.
6. Fine-tuning and preference tuning are the heaviest levers; justify them with enough high-quality labelled preference data that cheaper changes cannot address, and use SageMaker Ground Truth for the human labels.
7. Validate every change against a held-out evaluation set with a Bedrock evaluation job before rollout, because a fix for one cluster routinely regresses another.
8. Watch for feedback bias; the users who rate are not the whole population, so track quality across all interactions, not just the ones that left a signal.
9. Guardrails can enforce a rule independently of the prompt, which is more reliable than hoping a prompt edit holds for a safety-critical behaviour.
10. Feedback data is verbatim user content and can carry PII; govern every store with redaction, least-privilege access, and retention, and never train on raw un-redacted text.