Why this matters

Unit tests tell you your code runs. Eval gates tell you your AI system still behaves correctly. These are different things. A prompt change that passes all unit tests can silently halve your system's quality score. CI/CD with eval gates is the engineering practice that makes AI systems deployable with confidence rather than with hope.

Core concepts

Why unit tests are not enough for AI systems

Unit tests verify deterministic behavior: given input X, function returns Y. AI systems are not deterministic in that way. The behavior under test is quality: does the output satisfy the rubric? Does the classification accuracy stay above 85%? Does the retrieved context remain relevant?

Eval gates are a threshold-based quality check wired into the deployment gate. If quality drops below the threshold, the deployment is blocked.

The AI CI/CD pipeline shape

push / PR
  |
  v
lint (ruff, mypy)
  |
  v
unit tests (pytest)
  |
  v
eval gate (regression + quality thresholds)
  |
  v
deploy (only if eval passes)

The eval step sits between tests and deploy. It should be fast enough to run on every PR (under 5 minutes) and deterministic enough to not produce flaky failures.

GitHub Actions workflow with eval gate

# .github/workflows/ai_ci.yml
name: AI CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install ruff mypy
      - run: ruff check .
      - run: mypy src/

  test:
    needs: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v

  eval:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - name: Run eval gate
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python evals/run_ci_eval.py --threshold 0.80 --warn-threshold 0.85
      - name: Upload eval report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-report
          path: eval_report.json

  deploy:
    needs: eval
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - name: Deploy
        run: echo "Deploying..."  # replace with real deploy step

The eval gate script

# evals/run_ci_eval.py
import argparse
import json
import sys
from pathlib import Path


BLOCKING_THRESHOLD = 0.80   # fail the job below this
WARNING_THRESHOLD = 0.85    # warn but pass between this and blocking


def run_eval_gate(threshold: float, warn_threshold: float) -> None:
    golden = load_golden_set("evals/golden_set.jsonl")
    results, _ = run_regression(my_system_function, golden)

    total = len(results)
    passed = sum(1 for r in results if r.passed)
    pass_rate = passed / total if total else 0.0

    report = {"pass_rate": pass_rate, "passed": passed, "total": total}
    Path("eval_report.json").write_text(json.dumps(report, indent=2))

    if pass_rate < threshold:
        print(f"EVAL GATE FAILED: pass_rate={pass_rate:.2%} < threshold={threshold:.2%}", file=sys.stderr)
        sys.exit(1)
    elif pass_rate < warn_threshold:
        print(f"EVAL GATE WARNING: pass_rate={pass_rate:.2%} is below warn threshold={warn_threshold:.2%}")
    else:
        print(f"EVAL GATE PASSED: pass_rate={pass_rate:.2%}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--threshold", type=float, default=0.80)
    parser.add_argument("--warn-threshold", type=float, default=0.85)
    args = parser.parse_args()
    run_eval_gate(args.threshold, args.warn_threshold)

Blocking vs warning thresholds

Use two thresholds:

• Blocking threshold (e.g. 0.80) — below this, the deployment is blocked. The change broke something important.
• Warning threshold (e.g. 0.85) — above blocking but below this, the job passes but logs a warning. Worth reviewing before merging but not an emergency.

This prevents alert fatigue (every minor fluctuation blocks deploys) while still catching serious regressions.

Secrets management

API keys for evals go in GitHub repository secrets (ANTHROPIC_API_KEY), never in the repo. Access them via ${{ secrets.KEY_NAME }} in the workflow YAML. Consider using a dedicated low-quota key for CI evals to limit blast radius if the key is ever exposed.

Common mistakes

Running evals in the same job as tests — if an API call fails, you cannot tell whether the unit tests or the eval caused the job to fail. Keep them in separate jobs with clear names.

No artifact upload — the eval report tells you exactly which cases failed and why. If you do not upload it as a CI artifact, debugging a failed gate requires re-running locally.

Threshold too tight — setting the blocking threshold at 1.0 (100% pass rate) means any flaky LLM response blocks the deploy. Start at 0.80, tighten as the eval suite matures.

Hardcoding the threshold — pass thresholds as CLI arguments or environment variables so you can tighten them incrementally without modifying the script.

Try it yourself

Write a .github/workflows/ai_ci.yml that runs lint, tests, and an eval gate for a small AI function you own. The eval step should: load a golden set from evals/golden_set.jsonl, run the function against each case, compute pass rate, exit non-zero if pass rate is below 0.80, and upload eval_report.json as a CI artifact. Test it locally by running python evals/run_ci_eval.py --threshold 0.80 and verify the exit code matches the pass/fail result.