方寸 Portal

AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Implement A/B evaluation framework for prompt variants

Implement A/B Evaluation for Prompt Variants

Before shipping a prompt change you need evidence it improves the metric that matters, not just anecdotal inspection of a few examples.

What to build

Implement PromptABTest:

  1. Constructor: variant_a_fn, variant_b_fn (both Callable[[str], str]), score_fn: Callable[[str, str], float] (response + reference), pass_threshold=0.7.

  2. run(cases: list[dict]) -> dict — Each case has "input" and "reference". Returns {"a_avg_score", "b_avg_score", "score_delta" (B-A), "a_pass_rate", "b_pass_rate", "b_win_rate", "tie_rate", "a_win_rate", "recommendation"}. Recommendation: "deploy_b" if delta >= 0.05, "keep_a" if <= -0.05, else "inconclusive". Ties within 0.01.

  3. per_case_results(cases) -> list[dict] — Returns {"input", "a_score", "b_score", "winner"} per case.

Constraints

  • Standard library only.

Evaluation / medium / Step 22 of 36

Practice stage

Evaluation and review loops

Hint

Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.

Next drill
Build a regression test suite for LLM outputs
Success criteria
  • - Produces a useful signal, not decorative output
  • - Makes regression review easier
  • - Would support a benchmark or observability loop
Review checklist
  • - Would this output help decide what to fix next?
  • - Are important failure modes visible?
  • - Does the score hide any ambiguity I should record?
Related lessons
Evaluation metrics for LLM outputsBuilding eval harnesses

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.