The deployment risk for AI features
Deploying a new model version, prompt, or retrieval change is fundamentally different from deploying a bug fix. Code changes are verifiable: the test suite passes or fails. AI quality changes are probabilistic: average quality might improve while a specific failure mode gets worse. A new model version that improves average quality by 3% while degrading handling of edge cases by 20% is a risky trade that full-deployment-then-measure will not catch in time.
The solution is staged rollout with quality measurement at each stage.
Traffic splitting
Route a percentage of users to the new variant while the majority uses the current version. The split must be:
Deterministic: the same user must always get the same variant within an experiment. If user A sees the treatment on Monday, they must see it on Tuesday when you measure follow-up behavior. Use a hash of (experiment_id, user_id) modulo a bucket range.
Sticky across sessions: if assignment changes between the user's request and their follow-up action, your quality measurements are invalid.
Independent across experiments: two simultaneous experiments should not interfere. Hash separately for each experiment.
What to measure
Define quality metrics before starting the experiment:
| Metric type | Example |
|---|---|
| Outcome | User thumbs-up/down rate |
| Proxy quality | LLM-as-judge score on sampled responses |
| System | P95 latency, error rate |
| Business | Completion rate, follow-up queries |
Do not start with "we'll know it when we see it." Define pass/fail criteria before running.
Staged rollout stages
A typical safe rollout:
- Internal only (0% of users): deploy to your team's user accounts. Catch obvious breakage without user impact.
- Canary (1-5%): small production traffic sample. Monitor error rates and latency.
- Small ramp (10-20%): enough traffic to accumulate quality signal. Run for 24-48 hours minimum.
- Full rollout (100%): only after quality metrics confirm the variant meets or exceeds the baseline.
Each stage should have an explicit "stop condition": the criteria that will trigger an automatic rollback or pause.
Rollback plan
Always define rollback before starting. For prompt-based changes, rollback is an environment variable change (no code deployment). For model version changes, keep the previous model endpoint active during the experiment period.
Common mistakes
Testing with too few samples. LLM quality metrics are noisy. A sample of 30 responses is not enough to detect a 5% quality difference. You need at minimum 200-500 responses per variant before drawing conclusions.
Not controlling for time-of-day effects. If your canary runs Monday morning and your control runs Friday afternoon, you are measuring time-of-day differences, not model quality. Run both variants simultaneously.
Declaring a winner on latency alone. A faster model that produces worse outputs is not an improvement. Always include a quality metric alongside system metrics.
Not monitoring novelty effects. New features often see a temporary quality boost because users engage differently with unfamiliar UI. Monitor quality over at least one full week, not just the first day of the rollout.