Why this matters

AI features break in ways that standard test suites do not catch. A code change that does not touch any test can still degrade output quality because the underlying model was updated, a retrieved document changed, or a subtle interaction between prompt components changed behavior. CI/CD for AI requires layering quality gates on top of the standard test pipeline — not replacing it.

The engineering challenge: LLM outputs are non-deterministic and expensive to evaluate. You cannot run ten thousand evaluation cases on every commit. This lesson covers the patterns that give you meaningful quality signals within CI time and budget constraints.

Core concepts

The evaluation gate pattern

An eval gate is a CI step that runs a subset of your evaluation suite and blocks the merge if quality drops below a threshold. The gate should be:

• Fast enough for CI: run in under 5 minutes, meaning 50–200 cases, not the full thousand
• Representative: the cases should cover your most common failure modes, not just happy paths
• Deterministic enough: use temperature=0 for all CI evaluations so the same prompt gives the same output

import json
import sys
from pathlib import Path


def run_eval_gate(
    cases_path: Path,
    threshold: float = 0.85,
) -> bool:
    '''Return True if quality meets threshold. Exit with code 1 if not.'''
    cases = [json.loads(line) for line in cases_path.read_text().splitlines() if line.strip()]
    passed = 0

    for case in cases:
        result = evaluate_case(case)
        if result["passed"]:
            passed += 1
        else:
            print(f"FAIL [{case['id']}]: {result['reason']}")

    pass_rate = passed / len(cases) if cases else 0.0
    print(f"Eval gate: {passed}/{len(cases)} passed ({pass_rate:.1%})")

    if pass_rate < threshold:
        print(f"BLOCKED: pass rate {pass_rate:.1%} below threshold {threshold:.1%}")
        return False
    return True


if __name__ == "__main__":
    ok = run_eval_gate(Path("eval/ci_cases.jsonl"), threshold=0.85)
    sys.exit(0 if ok else 1)

In your CI pipeline:

# .github/workflows/ai-quality.yml
- name: Run eval gate
  run: python scripts/eval_gate.py
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Golden tests

Golden tests are input/expected-output pairs stored in version control. Unlike unit tests, the expected output is the actual model output from a known-good version. When output changes, the test fails — and you decide whether the change is an improvement or a regression.

import json
from pathlib import Path
from typing import NamedTuple


class GoldenCase(NamedTuple):
    id: str
    input: dict
    expected_output: str
    tolerance: float = 0.9  # semantic similarity threshold


def load_golden_cases(path: Path) -> list[GoldenCase]:
    cases = []
    for line in path.read_text().splitlines():
        if not line.strip():
            continue
        data = json.loads(line)
        cases.append(GoldenCase(**data))
    return cases


def update_golden(case_id: str, new_output: str, path: Path) -> None:
    '''Approve a new output as the golden reference.'''
    lines = path.read_text().splitlines()
    updated = []
    for line in lines:
        data = json.loads(line)
        if data["id"] == case_id:
            data["expected_output"] = new_output
        updated.append(json.dumps(data))
    path.write_text("\n".join(updated) + "\n")

The workflow: when a prompt change intentionally improves output, run python scripts/update_golden.py --id case-123 to approve the new output. This creates a diff in version control that reviewers can see and approve.

Prompt version control

Prompts are code. Store them in version-controlled files, not database strings or hardcoded f-strings. The pattern:

prompts/
  summarize_v1.txt        # previous version (keep for rollback)
  summarize_v2.txt        # current production version
  summarize_v3.txt        # candidate in review

Load the active version from configuration:

import os
from pathlib import Path


def load_prompt(name: str) -> str:
    version = os.environ.get(f"PROMPT_VERSION_{name.upper()}", "v2")
    prompt_path = Path("prompts") / f"{name}_{version}.txt"
    if not prompt_path.exists():
        raise FileNotFoundError(f"Prompt not found: {prompt_path}")
    return prompt_path.read_text().strip()

When you want to deploy v3, you update the environment variable. When you want to roll back, you update the environment variable again. No code deployment required.

Canary deployments for AI features

A canary deployment routes a small percentage of traffic to the new version while the majority continues to use the old version. For AI features, the canary phase serves two purposes: detecting technical failures and detecting quality degradation.

import random
from dataclasses import dataclass


@dataclass
class PromptCanaryConfig:
    canary_version: str
    canary_pct: float  # 0.0 to 1.0
    control_version: str


def select_prompt_version(config: PromptCanaryConfig, user_id: str) -> str:
    '''Deterministically assign users to canary or control based on user_id hash.'''
    # Use hash for sticky assignment — same user always gets same version
    user_bucket = hash(user_id) % 100 / 100
    if user_bucket < config.canary_pct:
        return config.canary_version
    return config.control_version

During the canary phase, compare quality metrics between the two groups. If the canary group shows no quality regression after N requests, promote it to 100%.

Feature flags for AI features

Feature flags let you decouple deploy from release. An AI feature that is deployed but flagged off can be enabled for internal users first, then gradually rolled out. This is especially important for AI features where you want to monitor production quality before full exposure.

import os


class AIFeatureFlags:
    '''Read feature flag state from environment. Override in tests.'''

    @staticmethod
    def is_enabled(flag_name: str, user_id: str | None = None) -> bool:
        env_key = f"FF_{flag_name.upper()}"
        raw = os.environ.get(env_key, "disabled")

        if raw == "enabled":
            return True
        if raw == "disabled":
            return False

        # Percentage rollout: "pct:25" enables for ~25% of users
        if raw.startswith("pct:") and user_id is not None:
            pct = float(raw.split(":")[1]) / 100
            return (hash(f"{flag_name}:{user_id}") % 100) / 100 < pct

        return False

Working example

A complete CI workflow that runs unit tests, then the eval gate, then updates golden references on approval:

name: AI Feature Quality Gate

on: [pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - run: pip install -r requirements-dev.txt
      - run: pytest tests/ -x -q

  eval-gate:
    needs: unit-tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - run: pip install -r requirements.txt
      - name: Run eval gate (CI subset, temperature=0)
        run: python scripts/eval_gate.py --cases eval/ci_cases.jsonl --threshold 0.85
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PROMPT_VERSION_SUMMARIZE: ${{ github.event.pull_request.head.sha }}

Common mistakes

No eval gate, only unit tests. Unit tests confirm that your code runs. They do not confirm that your LLM outputs meet quality standards. Add at least 20–30 representative eval cases to your CI pipeline.

Running the full eval suite in every CI run. A 5000-case eval suite at $0.002/case costs $10 per run. With 50 PRs per week that is $500/week. Use a representative 100-case CI subset and run the full suite nightly.

Storing prompts as database strings without version control. A prompt change made in a database has no reviewer, no diff, and no rollback path. Prompts stored as files get all the benefits of your existing version control workflow.

Not using deterministic settings in CI. temperature=1.0 in CI will cause golden tests to flap randomly. Always use temperature=0 and seed parameters when available in CI evaluation runs.

Canary deployments without quality metrics. A canary that only monitors error rates and latency will not catch output quality regression. Define and track at least one quality metric during the canary phase.

Try it yourself

1. Create a prompts/ directory in one of your projects and move at least one hardcoded prompt into a versioned file. Load it with os.environ.get("PROMPT_VERSION_...").
2. Write five golden test cases for an LLM feature you own. Store them as JSONL. Write a test that loads them and checks semantic similarity between expected and actual output.
3. Implement the select_prompt_version function above with sticky user assignment. Verify that the same user ID always returns the same version across multiple calls.