Golden Dataset Curation and Maintenance
A golden dataset is not a one-time artifact — it is a living document that needs the same maintenance discipline as your application code. The teams that treat it as permanent tend to regress silently as the dataset drifts away from the real distribution.
What belongs in a golden dataset
A golden dataset should cover the distribution your feature actually encounters, not the distribution you wish it encountered. Three sources produce high-quality seed cases:
Production failures. When your monitoring surfaces a bad response — a user thumbs-down, a faithfulness score below 0.5, a judge that flagged hallucination — that case should enter the golden dataset immediately. These are guaranteed to be difficult and guaranteed to be real. Log the original query, the retrieved context, the generated response, and what made it fail.
Sampled production traffic. A random 0.1% sample of production queries, labeled by human reviewers, gives you cases representative of real usage. Without this, your dataset will skew toward edge cases and miss the "common but subtle" failures.
Deliberately constructed edge cases. Cases you construct to test specific behaviors: boundary conditions, ambiguous queries, multi-hop reasoning, off-topic requests, queries in under-represented languages. These test hypotheses about where the feature might fail.
How many cases you need
| Goal | Minimum cases |
|---|---|
| Catch major regressions (>10% pass rate drop) | 30 |
| Detect moderate regressions (>5% drop) | 100 |
| Meaningful category-level breakdowns | 20 per category |
| Statistically significant A/B comparison | 50-100 per variant |
More is not always better. A 10,000-case dataset that takes 90 minutes to run on every PR will be skipped. Keep the core golden set at 50-200 cases and run it on every commit. Maintain a larger "extended" set that runs nightly.
The labeling process
Good labels require a written rubric before anyone labels anything. The rubric defines what each score value means for this specific task, not in the abstract. Example rubric for a RAG Q&A feature:
- 1.0 (Pass): All claims in the answer are directly supported by retrieved chunks. The answer addresses the user's question. No relevant information from the context was omitted.
- 0.5 (Borderline): Answer is mostly correct but makes one unsupported claim, or the question is addressed but incompletely.
- 0.0 (Fail): Answer contains a fabricated fact not present in any context chunk, or the question is not addressed.
Two reviewers should independently label a 20% sample. Compute inter-rater agreement (Cohen's Kappa). If Kappa < 0.6, the rubric is ambiguous — resolve disagreements, update the rubric, re-label. Do not proceed with a rubric that humans cannot consistently apply.
Dataset rot and maintenance
Golden datasets rot in two ways:
Distribution rot. User behavior changes. A golden set seeded from 2023 traffic may not represent 2025 queries. Audit quarterly: compare the query types in your golden set to a recent 1,000-query sample. Add cases from underrepresented patterns, retire cases from patterns that no longer appear.
Label rot. Your feature evolves, so "correct" outputs change. A response you labeled 1.0 six months ago may now be borderline because the product added a requirement for citation links. Re-label at least the failure cases whenever the feature's definition of "correct" changes.
Versioning
Version your golden dataset the same way you version your code. Store it in source control as a JSONL file. Every case has a case_id, version_added, and optionally version_retired. When you retire a case, do not delete it — mark it "active": false. This lets you reconstruct historical pass rates accurately.
The maintenance cadence
- Continuous: add new cases from production failures within 24 hours of discovery
- Per-deploy: run the golden set, store results with commit hash
- Monthly: review coverage report, seed from recent production sample, update rubric if needed
- Quarterly: full audit — inter-rater agreement check, distribution comparison, retire stale cases