AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Implement A/B testing for model versions

Implement A/B Testing for Model Versions

When you want to test a new model version or prompt in production, rolling it out to all users at once is risky. A/B testing lets you route a fraction of traffic to the candidate version, collect quality metrics, and compare against the control before committing to a full rollout.

What to build

Implement an ABTestRouter that:

Takes a control_config and candidate_config (each with version_id: str).
assign(user_id: str, test_id: str) -> str — deterministically returns either "control" or "candidate". The same user always gets the same assignment for the same test (sticky assignment via hash).
record_outcome(test_id: str, variant: str, metric_name: str, value: float) -> None — record a quality metric for a variant.
get_results(test_id: str) -> ABTestResults — return aggregated results for both variants.
ABTestResults has per-variant n and mean for each recorded metric.
should_promote(test_id: str, metric: str, min_improvement_pct: float = 5.0) -> bool — return True if candidate outperforms control by at least min_improvement_pct percent.

Constraints

Standard library only.
Candidate percentage is configurable (default 10% of traffic).
Sticky assignment must be deterministic: same user_id + test_id always returns same variant.

Evaluation / medium / Step 14 of 36

Practice stage

Evaluation and review loops

Hint

Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.

Next drill

Build an LLM call tracer with span export

Success criteria

- Produces a useful signal, not decorative output
- Makes regression review easier
- Would support a benchmark or observability loop

Review checklist

- Would this output help decide what to fix next?
- Are important failure modes visible?
- Does the score hide any ambiguity I should record?

from __future__ import annotations
import hashlib
from collections import defaultdict
from dataclasses import dataclass, field

@dataclass
class VariantConfig:
    version_id: str
    extra_params: dict = field(default_factory=dict)

@dataclass
class VariantStats:
    n: int
    mean: float

@dataclass
class ABTestResults:
    test_id: str
    control: dict[str, VariantStats]
    candidate: dict[str, VariantStats]

class ABTestRouter:
    def __init__(
        self,
        control_config: VariantConfig,
        candidate_config: VariantConfig,
        candidate_pct: float = 0.10,
    ):
        # TODO: store configs and pct
        raise NotImplementedError

def assign(self, user_id: str, test_id: str) -> str:
        # TODO: deterministically assign "control" or "candidate"
        raise NotImplementedError

def record_outcome(self, test_id: str, variant: str, metric_name: str, value: float) -> None:
        # TODO: store metric value for this variant
        raise NotImplementedError

def get_results(self, test_id: str) -> ABTestResults:
        # TODO: aggregate per-variant metrics
        raise NotImplementedError

def should_promote(self, test_id: str, metric: str, min_improvement_pct: float = 5.0) -> bool:
        # TODO: return True if candidate outperforms control by min_improvement_pct
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.