Your personal operating system for career transition.

Implement A/B Evaluation for Prompt Variants

Changing a prompt without measuring the impact is one of the most common mistakes in applied AI engineering. Build a rigorous A/B evaluation pipeline that compares two prompt variants on the same golden dataset.

What you are building

Implement an ABEvaluator that:

1. Runs both variants -- for each test case, generates responses using variant_a_fn and variant_b_fn.
2. Scores both responses -- using a provided score_fn(response, expected) -> float.
3. Computes win/loss/tie statistics -- a "win" for B is when B's score exceeds A's by more than margin (default 0.05).
4. Identifies regressions -- any case where B scores more than 0.10 lower than A.
5. Returns a recommendation -- "b" if win rate > 55% AND regression rate <= 5%, "a" if A win rate > 55%, else "inconclusive".

Constraints

• Use only the Python standard library.
• Both variants receive the same input for each case.
• cases is a list of dicts with case_id, input, expected.