Building eval harnesses

Why this matters

A prompt change that improves 80% of cases but breaks the 20% you care about most is not an improvement -- it is a regression you did not catch. Eval harnesses are the infrastructure that turns 'I think this is better' into 'I can prove this is better on the cases that matter, and nothing important got worse.'

For engineers coming from web development, this is analogous to a test suite. The difference is that LLM outputs are non-deterministic and the 'correct' answer is often a spectrum rather than a boolean. The discipline is the same: define what correct means before you change anything, then measure it after.

Core concepts

Golden datasets. A golden dataset is a curated collection of test cases where you have already decided what the correct output looks like. Each case has: an input (the user query or prompt), a reference output (the expected response), and metadata (category, difficulty, known edge cases). 'Golden' signals that these cases are trusted -- they represent the behaviors you are not willing to regress on.

Characteristics of a good golden dataset:

• Representative: covers the real distribution of inputs your feature receives, not just easy cases
• Challenging: includes edge cases, ambiguous queries, and cases where naive approaches fail
• Balanced: has enough examples of each important category to detect category-level regressions
• Small enough to run fast: 50-200 cases is usually enough to catch regressions without making CI slow

Regression testing. Once you have a golden dataset, every significant prompt change, model swap, or configuration update should run the full eval suite before shipping. A regression is any case that passed before and fails after. Track the regression rate as a deployment gate.

A/B evaluation. Head-to-head comparison between two prompt variants on the same dataset. Report: win rate (variant B won more cases than A), tie rate, and loss rate. A 52% win rate on 50 cases is noise, not signal.

Eval harness architecture. A production eval harness has four layers:

1. Dataset loader: reads cases from a file or database, applies filters
2. Generation layer: calls the LLM feature under test
3. Scoring layer: applies metrics (deterministic, LLM judge, or both) to each response
4. Reporting layer: aggregates scores, compares to baseline, flags regressions

Integrating with CI. Run the golden set on every pull request. Gate merges on the regression threshold. Store results with the git commit hash, model version, and timestamp.

Working example

A complete eval harness for a Q&A feature:

import json
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable


@dataclass
class EvalCase:
    case_id: str
    question: str
    reference_answer: str
    category: str
    context_chunks: list[str] = field(default_factory=list)


@dataclass
class RunResult:
    case_id: str
    category: str
    response: str
    score: float
    passed: bool
    latency_ms: int


class EvalHarness:
    def __init__(
        self,
        dataset_path: str,
        generate_fn: Callable[[str, list[str]], str],
        score_fn: Callable[[str, str, list[str]], float],
        pass_threshold: float = 0.7,
        regression_threshold: float = 0.05,
    ):
        self.dataset_path = Path(dataset_path)
        self.generate_fn = generate_fn
        self.score_fn = score_fn
        self.pass_threshold = pass_threshold
        self.regression_threshold = regression_threshold

    def load_cases(self, categories: list[str] | None = None) -> list[EvalCase]:
        cases = []
        with open(self.dataset_path) as f:
            for line in f:
                raw = json.loads(line)
                case = EvalCase(**raw)
                if categories is None or case.category in categories:
                    cases.append(case)
        return cases

    def run(self, cases: list[EvalCase]) -> list[RunResult]:
        results = []
        for case in cases:
            t0 = time.monotonic()
            response = self.generate_fn(case.question, case.context_chunks)
            latency_ms = int((time.monotonic() - t0) * 1000)
            score = self.score_fn(response, case.reference_answer, case.context_chunks)
            results.append(RunResult(
                case_id=case.case_id, category=case.category, response=response,
                score=score, passed=score >= self.pass_threshold, latency_ms=latency_ms,
            ))
        return results

    def report(self, results: list[RunResult], baseline: dict | None = None) -> dict:
        total = len(results)
        passed = sum(1 for r in results if r.passed)
        avg_score = sum(r.score for r in results) / total if total else 0.0
        avg_latency = sum(r.latency_ms for r in results) / total if total else 0.0
        by_category: dict[str, list[float]] = {}
        for r in results:
            by_category.setdefault(r.category, []).append(r.score)
        rep = {
            'total': total,
            'pass_rate': passed / total if total else 0.0,
            'avg_score': avg_score,
            'avg_latency_ms': avg_latency,
            'by_category': {cat: sum(s)/len(s) for cat, s in by_category.items()},
            'failures': [r.case_id for r in results if not r.passed],
            'by_case': {r.case_id: r.score for r in results},
        }
        if baseline:
            delta = avg_score - baseline.get('avg_score', avg_score)
            rep['vs_baseline'] = {'score_delta': delta, 'is_regression': delta < -self.regression_threshold}
        return rep

Usage in CI:

harness = EvalHarness(dataset_path='eval/golden_set.jsonl',
                      generate_fn=my_qa_feature, score_fn=llm_faithfulness_score)
cases = harness.load_cases()
results = harness.run(cases)
report = harness.report(results, baseline=load_baseline_report())
if report.get('vs_baseline', {}).get('is_regression'):
    raise SystemExit('Eval regression detected. Review before merging.')

Common mistakes

1. Building the golden set from easy cases only. If your dataset is all simple queries the model already handles well, regression detection is useless. Actively seed with hard and edge cases.

2. Not storing eval results. Running evals without persisting results means you cannot trend over time. Store results with the git commit hash, model version, and timestamp.

3. Treating the eval suite as a one-time artifact. Eval datasets go stale. Add new cases when users report failures.

4. Using the same LLM to generate and judge. Self-judging inflates scores. Use a different model, or a deterministic scorer, for at least half your eval metrics.

5. No latency tracking in the harness. Prompt changes that improve quality but double latency are not free improvements.

Try it yourself

Build a golden dataset of 20 test cases for a hypothetical document Q&A feature. Each case should have a question, a reference answer, a context chunk, and a category label. Then implement a run_regression_check function that takes two sets of eval results (baseline and current), identifies any case where the current score is more than 0.15 lower than the baseline score, and returns those cases with their delta.