AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Build a regression test suite for LLM outputs

Build a Regression Test Suite for LLM Outputs

A regression suite catches when a change makes previously-passing cases fail. It focuses on non-negotiable behaviors the system has already demonstrated correctly.

What to build

Implement RegressionSuite:

add_case(case_id, input, expected_patterns, forbidden_patterns=[], tags=[]) — expected_patterns must appear (case-insensitive). forbidden_patterns must not appear.
run(generate_fn: Callable[[str], str]) -> dict — Returns {"passed", "failed", "total", "pass_rate", "failures", "by_case"}. failures is a list of {"case_id", "missing_patterns", "found_forbidden"}. by_case is {case_id: bool}.
diff_against_baseline(current, baseline) -> dict (static method) — Compares by_case dicts. Returns {"new_failures", "new_passes", "unchanged"}.
filter_by_tag(tag) -> "RegressionSuite" — Returns new suite with only matching-tag cases.

Constraints

Standard library only.

Evaluation / hard / Step 23 of 36

Practice stage

Evaluation and review loops

Hint

Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.

Next drill

Implement structured logging for AI call traces

Success criteria

- Produces a useful signal, not decorative output
- Makes regression review easier
- Would support a benchmark or observability loop

Review checklist

- Would this output help decide what to fix next?
- Are important failure modes visible?
- Does the score hide any ambiguity I should record?

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.