Building a Golden Dataset for LLM Evaluation

The situation

A team ships a Bedrock-backed assistant that answers customer questions from an internal knowledge base. It works, mostly. Then someone proposes swapping the underlying model for a cheaper one, and the room splits: half the team says quality will drop, half says it will hold, and nobody can settle it because there is no measurement. The last three prompt changes were merged on the strength of “looks better to me” and a couple of hand-picked examples that happened to be open in a tab.

The pattern repeats every time anything changes. A retrieval tweak that helps five questions someone remembers might be quietly breaking fifty they do not. A prompt edit that fixes a complaint about tone might have loosened a refusal that used to hold. Each change is argued from anecdote, and because the anecdotes are chosen after the change, they flatter it. The team has no way to say, in a number that means the same thing this week as last, whether the assistant got better or worse.

What they are missing is a fixed set of inputs with agreed-upon right answers: a golden dataset. Everything downstream, the model choice, the prompt library, the RAG configuration, is only as measurable as this dataset is representative. Building it well is the whole game, because a biased or thin eval set does not just fail to catch regressions, it actively certifies them.

What actually matters

A golden dataset, also called a ground-truth set, is a collection of representative inputs each paired with an accepted answer or, where a single answer is too rigid, a set of acceptance criteria the response has to satisfy. It is the reference the system is measured against. The value of every evaluation you ever run is capped by how honestly this set reflects what real users actually send, so the properties that make it trustworthy matter more than any tooling choice that comes later.

The first property is representativeness of the real distribution. The dataset has to look like production traffic, not like the questions that are easy to write. That means the common, boring middle in roughly the proportion it actually occurs, plus deliberate coverage of the cases that break systems: edge cases, ambiguous phrasing, long inputs, adversarial inputs, and the known-hard questions the team already knows are shaky. Skew the set toward tidy questions and you get an eval that reports high scores while real users hit the failures the set never sampled.

The most-missed slice is the questions the system should not answer. A trustworthy assistant refuses when the knowledge base does not cover something, when the request is out of scope, or when answering would mean inventing a fact. If the golden dataset contains only answerable questions, you can never measure appropriate refusal, and a model that confidently hallucinates on unanswerable inputs scores identically to one that correctly declines. Known-unanswerable cases, with “should refuse” as their accepted answer, are what let you catch the failure mode that hurts users most.

Then there is provenance and bias in how the examples are sourced. Real traffic and logs give you the true distribution but need scrubbing and labelling. Subject-matter experts give you authoritative answers and can invent the rare-but-critical cases logs have not seen yet. Synthetic generation, using a model to produce test inputs, is fast and cheap and fills gaps, but a set built only from synthetic data inherits the generating model’s blind spots and phrasing habits, so it measures how well you handle questions a model would ask rather than questions a human would. Synthetic examples belong in the mix, not as the whole of it.

Labelling is where subjective quality gets pinned down. For clear-cut tasks the accepted answer is obvious, but for tone, helpfulness, and “is this answer actually correct and complete” you need human judgement, applied consistently by people who know the domain. This is what a service like Amazon SageMaker Ground Truth is for: routing examples to human labellers or reviewers against a written rubric, so the labels are consistent rather than one engineer’s mood on a Tuesday.

Finally, the dataset is only reusable if it is disciplined over time. You need a held-out slice you never look at while tuning, or you will overfitOverfittingWhen a model stops learning the general pattern in your data and starts memorising the individual examples. prompts to the eval and get a number that flatters itself. And you need the whole thing versioned, so a score from this month and a score from last month are comparing the same yardstick rather than a quietly-edited one.

What we’ll filter on

1. Distribution match: does the set mirror real traffic, including the boring common cases in their real proportion?
2. Hard and edge coverage: are known-hard, ambiguous, and adversarial inputs deliberately included?
3. Refusal coverage: are known-unanswerable and out-of-scope questions present, labelled as “should refuse”?
4. Sourcing balance: does it draw from real logs and experts, not synthetic generation alone?
5. Label quality: is subjective quality labelled by domain humans against a consistent rubric?
6. Reusability discipline: is there a protected held-out split, and is the dataset versioned for comparable results over time?

The dataset landscape

Sourcing from real traffic and logs. Mine production requests, or pre-launch pilot logs, for genuine inputs. This is the truest picture of the distribution and surfaces phrasings nobody would have invented. The cost is effort: logs need de-duplicating, scrubbing of personal data, and answers attached, because a raw request has no accepted answer until someone supplies one. It also cannot cover cases that have not happened yet, which is where experts and synthetic generation come in.

Subject-matter experts. Domain people write both the inputs and the authoritative answers, and crucially they can author the rare, high-stakes cases that logs rarely contain: the compliance edge, the dangerous misunderstanding, the question that must be refused. Expert time is the scarce resource, so spend it on the hard and high-consequence slices rather than the common middle that logs already cover well.

Synthetic generation. Use a model to generate candidate inputs, paraphrases, and edge variations at volume. Excellent for widening coverage cheaply, stress-testing with adversarial phrasings, and filling thin categories. The trap is building the set only this way: synthetic-only data carries the generator’s stylistic and topical bias, over-represents what the model finds natural, and can miss the awkward, misspelt, half-formed way real people actually write. Treat synthetic examples as a supplement that a human reviews, never as the ground truth itself.

Human labelling with SageMaker Ground Truth. However inputs are sourced, the accepted answers and quality judgements for anything subjective need consistent human labelling. Ground Truth manages labelling and review workflows against a rubric, whether the workforce is your own team, a vendor, or a managed one, and gives you consistent labels instead of ad-hoc opinions. This is the mechanism that turns “we think this answer is good” into a repeatable label other people would agree with.

Stratification and sizing. Rather than one undifferentiated pile, divide the set into strata: topic areas, difficulty bands, input lengths, answerable versus should-refuse. Sizing follows from wanting each stratum big enough that a change moving it is visible above noise, so a small but deliberately stratified set that covers every category beats a large set that is ninety per cent easy questions. Report scores per stratum, not just one blended average, because a headline number can hold steady while refusal quietly collapses underneath it.

Held-out split and versioning. Partition the set into a development slice you iterate against and a held-out slice you check only occasionally, to catch prompts that have been overfit to the visible eval. And version the whole dataset, in source control or a data store, so every result is tagged with the dataset version that produced it and month-over-month comparisons are honest. An eval set edited in place, with no version, silently invalidates every historical number.

Bedrock evaluation jobs and an LLM-as-a-judgeLLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. harness. The dataset is the input to whatever runs the scoring. Amazon Bedrock offers model evaluation jobs (automatic metrics, human review, or an LLM acting as judge) and RAG evaluation for retrieval-augmented setups, all of which take a curated prompt dataset and score responses against it. Alternatively you run your own harness where a strong model grades each response against the accepted answer or rubric. Either way the golden dataset is what is being fed in; the harness is interchangeable, the dataset is the asset.

Side by side

Source or practice	Distribution fidelity	Covers hard and refusal cases	Bias risk	Effort or cost	Best role
Real traffic and logs	✓	✗ (only what happened)	Low	High (scrub, label)	The backbone of the set
Subject-matter experts	✗ (curated)	✓	Low	High (expert time)	Rare, hard, high-stakes cases
Synthetic generation	✗	✓ (breadth)	High if used alone	Low	Widen coverage, fill gaps
Ground Truth labelling	n/a	n/a	Low (consistent)	Medium	Consistent subjective labels
Stratification and sizing	✓	✓	Low	Low	Make every category visible
Held-out split	n/a	n/a	Low	Low	Catch overfitting to the eval
Versioning	n/a	n/a	Low	Low	Comparable results over time

Reading the table: no single source builds the set. Logs give the distribution, experts and synthetic generation give the hard and refusal coverage logs lack, Ground Truth makes the labels consistent, and stratification, a held-out split, and versioning are what make the result trustworthy and reusable rather than a one-off snapshot.

The picks in depth

Start from real traffic and build the backbone from the true distribution. Pull a sample of production or pilot requests, scrub anything sensitive, and stratify what remains by topic and difficulty so you can see the shape of what users actually ask. This anchors the set in reality and stops the common failure of an eval made entirely of questions the team found interesting to write. Attach an accepted answer or acceptance criteria to each one; a logged input without a label is a test case with no pass condition.

Layer in experts and synthetic generation to cover what logs cannot. Have domain experts author the rare-but-critical cases and, importantly, the known-unanswerable and out-of-scope questions with “should refuse” as the accepted answer, because that slice is how you measure whether the system declines instead of inventing. Use synthetic generation to multiply coverage: paraphrase real questions, generate adversarial variants, and fill thin strata. Keep a human in the loop reviewing synthetic examples, and never let the set tip to synthetic-only, or you measure the generator’s world rather than your users.

Label subjective quality consistently, and protect a held-out slice. For anything where “correct” is a judgement (tone, completeness, factual accuracy against the knowledge base), route examples through a labelling workflow like SageMaker Ground Truth with a written rubric, so two labellers reach the same verdict. Then split off a held-out portion you do not look at while tuning prompts. Iterate against the development slice; check the held-out slice only now and then. When the two diverge, the visible eval has been overfit and its scores have stopped meaning anything general.

Version the dataset and feed it into a repeatable harness. Store the set under version control or in a data store with an explicit version tag, and record which version produced every score, so a comparison across months is honest rather than a comparison of two different yardsticks. Point it at Amazon Bedrock model evaluation or RAG evaluation jobs, or at your own LLM-as-a-judge harness that grades each response against the accepted answer. The scoring mechanism can change; the dataset is the durable asset, and a model swap becomes a measured decision instead of an argument.

A worked example: settling the model swap

The team that could not agree on the cheaper model builds a golden set before touching anything. They pull 400 real questions from three months of pilot logs, scrub them, and stratify: 60 per cent common account and product questions in their real proportion, 20 per cent known-hard cases (multi-part questions, ambiguous phrasing, long inputs), and 20 per cent that should refuse (out-of-scope requests, questions the knowledge base does not cover, and a few adversarial “just make something up” inputs).

Experts label the accepted answers, and the should-refuse cases get “declines and does not invent a fact” as their pass condition. Ground Truth keeps the quality labels consistent. They version the set as v1 and split off 80 questions as held-out.

Now the swap is a measurement, not a debate. They run both models against the development slice through a Bedrock evaluation job and read the scores per stratum. The cheaper model matches on the common middle and the known-hard cases, within noise. But on the should-refuse stratum it drops sharply: it answers questions it should decline, inventing knowledge-base facts that do not exist. The blended average barely moved, which is exactly why the stratified breakdown mattered; a single number would have waved the swap through. They confirm the pattern holds on the held-out slice, then keep the current model for the refusal-sensitive paths and move on with a decision nobody has to argue about again.

What’s worth remembering

1. A golden dataset is representative inputs paired with accepted answers or acceptance criteria; every downstream evaluation is only as trustworthy as this set is honest.
2. Match the real distribution, including the boring common cases in their real proportion, not just the questions that were easy to write.
3. Deliberately cover known-hard, ambiguous, and adversarial inputs; an eval made of tidy questions certifies the failures it never sampled.
4. Include known-unanswerable and out-of-scope cases labelled “should refuse”, or you can never measure appropriate refusal, and a confident hallucinator scores the same as a careful decline.
5. Source from real logs and subject-matter experts as well as synthetic generation; a synthetic-only set inherits the generating model’s blind spots and phrasing.
6. Use human labelling with a consistent rubric, via a service like SageMaker Ground Truth, to pin down subjective quality like tone and correctness.
7. Stratify and report per stratum; a blended average can hold steady while refusal quietly collapses underneath it.
8. Keep a held-out slice you do not tune against, so you catch prompts overfit to the visible eval.
9. Version the dataset and tag every result with the version that produced it, so comparisons over time are honest.
10. Feed the set into Bedrock model or RAG evaluation jobs, or your own LLM-as-a-judge harness; the scoring mechanism is interchangeable, the golden dataset is the durable asset.