Your personal operating system for career transition.

Regression Testing Discipline for LLM-Powered Features

The hardest problem in AI engineering is not building the feature — it is ensuring that every change you make does not silently break something that was already working. Regression testing closes that loop.

Why AI regression testing is harder than unit testing

A unit test is deterministic. Run it twice, get the same result. An LLM feature test is probabilistic: the same prompt on the same model may produce different outputs on different calls, but the quality distribution should remain stable.

This means AI regression tests need to:

1. Tolerate small output variation while detecting meaningful quality drops
2. Define "passing" as a score above a threshold, not an exact string match
3. Aggregate across cases to distinguish real degradation from random noise

The golden dataset

The foundation is a curated dataset of test cases where you know what good looks like. Build it with these principles:

Seed from real failures. Every time a production request fails in a way that should not, add it to the golden dataset. This ensures the dataset tracks real failure modes rather than hypothetical ones.

Include adversarial cases. Cases that expose known weaknesses — ambiguous queries, requests at the edge of the feature scope, inputs likely to cause hallucination — are the most valuable. Easy cases that always pass add no regression detection value.

Keep it small enough to run fast. A 200-case dataset that runs in 3 minutes will actually get run. A 5,000-case dataset that takes 2 hours will be skipped. Start with 50-100 cases and grow deliberately.

Include category labels. Tag each case with a category (factual lookup, multi-hop reasoning, safety boundary, format compliance). Per-category scores surface regressions that aggregate metrics hide.

What to put in each test case

{
  "case_id": "multi-hop-001",
  "input": "What was the annual revenue growth rate between 2022 and 2023?",
  "reference": "Revenue grew 23% from $1.2B in 2022 to $1.48B in 2023",
  "context_chunks": ["...", "..."],
  "category": "multi-hop",
  "expected_patterns": ["23%", "1.2", "1.48"],
  "forbidden_patterns": ["I don't know", "cannot determine"],
  "min_faithfulness_score": 0.8
}

Each case should have at minimum: the input, a reference answer, and at least one of: expected patterns, a minimum score threshold, or a score function.

CI integration

The regression suite should run automatically on every pull request that touches prompt code, model configuration, retrieval logic, or embedding models. Gate merge on the regression threshold.

# .github/workflows/eval.yml
- name: Run eval suite
  run: python scripts/run_eval.py --dataset eval/golden_set.jsonl --baseline eval/baseline.json
- name: Check quality gate
  run: python scripts/check_gate.py --report eval/latest_report.json

Store the baseline report in version control alongside the code. When you intentionally improve a metric, update the baseline explicitly as part of the commit.

The triage workflow

When regressions are detected:

1. Identify the regression cases — which specific case_ids dropped?
2. Inspect the response — what did the model actually return on the failing case?
3. Classify the failure mode — hallucination? Retrieval miss? Format violation? Model refusal?
4. Trace to the cause — what changed? Prompt? Model? Retrieval logic? Index content?
5. Fix and re-run — verify the fix resolves the regression without introducing new ones

The triage steps are predictable, which means they can be documented before anything breaks. Write the runbook before the first production incident.

Updating the dataset

The golden dataset is a living artifact. Add new cases when:

• A production failure reveals a category not covered by the current dataset
• A feature expansion adds new input types
• An adversarial test finds a new failure mode

Remove cases when:

• The feature scope changes and a category is no longer relevant
• A case tests implementation details rather than observable behavior

Archive removed cases rather than deleting them — they may be relevant again after future changes.

The discipline mindset

Regression testing only works as a practice, not a one-time setup. The teams that do it well treat the golden dataset as a first-class artifact: it gets reviewed in code review, updated alongside feature changes, and its pass rate is tracked in the same dashboard as production quality metrics.

The teams that skip it discover their regressions in production, usually from user complaints.