Your personal operating system for career transition.

Implement A/B Testing for Model Versions in Production

Deploying a new model version without traffic splitting is risky. A/B testing routes a percentage of users to a new variant so you can measure quality before full rollout.

What to build

ABTestRouter(experiment_id: str, variants: list[Variant]):

• Variant: name, model, prompt_template, traffic_pct (sum must be 1.0)
• assign_variant(user_id) -> Variant — deterministic: same user always gets same variant
• record_outcome(user_id, variant_name, score, latency_ms)
• get_results() -> dict — per-variant: count, avg_score, avg_latency_ms, pass_rate (score >= 0.7)
• significant_winner(min_samples=30) -> str | None — winner if leading by 0.05+ avg_score with enough samples

Constraints

• Standard library only.
• Hash-based deterministic assignment.
• traffic_pct sum within 0.001 of 1.0.