Your personal operating system for career transition.

Create a Golden Dataset Test Suite

A golden dataset is a curated set of test cases you trust to detect regressions. Build the infrastructure to load, run, and report on a golden evaluation suite.

What you are building

Implement a GoldenSuite class that:

1. Loads cases from JSONL -- each line has case_id, input, expected, and category fields.
2. Runs evaluation -- for each case, calls a generate_fn and a score_fn, records score and pass/fail.
3. Detects regressions -- accepts a baseline report dict and identifies any case where current score is more than 0.10 below the baseline score for the same case_id.
4. Reports by category -- returns average score and pass rate broken down by category.
5. Outputs a portable report -- returns a dict with by_case scores suitable as the next run's baseline.

Constraints

• Use only the Python standard library.
• score_fn signature: (response: str, expected: str) -> float returning 0.0 to 1.0.
• generate_fn signature: (input: str) -> str.
• Pass threshold is configurable (default 0.7).
• Regression threshold is configurable (default 0.10).