Why this matters

Testing AI applications is genuinely harder than testing conventional software. LLM outputs are non-deterministic: the same prompt can return different phrasings on different runs. Provider APIs are expensive and rate-limited. Evaluation criteria are often qualitative. Engineers who apply conventional unit testing patterns either write tests so loose they catch nothing or so strict they break on every model upgrade.

This lesson covers a layered testing strategy: mock provider calls for fast unit tests, snapshot structural properties for regression testing, and use golden datasets for eval-driven quality gates.

Core concepts

Why conventional unit tests are insufficient

• Non-determinism: assert response == expected_string fails whenever phrasing varies
• Slow feedback: a 2-second LLM call per test case makes a 100-test suite take 3 minutes
• Cost: running tests against live APIs in CI costs money and consumes rate limit budget
• No ground truth: unlike a math function, there is no single correct output to assert against

The solution is layers: mock the provider for unit tests, test structure not content, and use real LLM calls only in scheduled eval runs.

Mocking LLM providers for unit tests

Mock at the HTTP boundary, not at a high abstraction level:

import pytest
from unittest.mock import AsyncMock, MagicMock, patch


def make_anthropic_response(text: str) -> MagicMock:
    msg = MagicMock()
    msg.content = [MagicMock(text=text)]
    msg.stop_reason = "end_turn"
    msg.usage.input_tokens = 100
    msg.usage.output_tokens = len(text.split())
    return msg


@pytest.fixture
def mock_anthropic():
    with patch("myapp.llm.anthropic_client") as mock_client:
        mock_client.messages.create = AsyncMock(
            return_value=make_anthropic_response(
                '{"entities": ["Alice", "Bob"], "sentiment": "positive", "confidence": 0.9}'
            )
        )
        yield mock_client


@pytest.mark.asyncio
async def test_extraction_returns_structured_output(mock_anthropic):
    from myapp.extraction import extract_entities

    result = await extract_entities("Alice and Bob had a great meeting.")

    assert result is not None
    assert "Alice" in result.entities
    assert result.sentiment == "positive"
    assert 0.0 <= result.confidence <= 1.0
    mock_anthropic.messages.create.assert_called_once()
    call_kwargs = mock_anthropic.messages.create.call_args.kwargs
    assert "messages" in call_kwargs

This test runs in milliseconds, costs nothing, and verifies extraction logic — not the LLM's phrasing.

Testing output structure, not content

For non-deterministic outputs, assert structure and type properties instead of exact string equality:

from pydantic import BaseModel


class SummaryOutput(BaseModel):
    headline: str
    key_points: list[str]
    word_count: int


def assert_valid_summary(output: SummaryOutput, min_points: int = 2) -> None:
    assert len(output.headline) > 0, "Headline must not be empty"
    assert len(output.headline) <= 200, "Headline must be concise"
    assert len(output.key_points) >= min_points, f"Expected at least {min_points} key points"
    assert all(len(p) > 0 for p in output.key_points), "Key points must not be empty"
    assert output.word_count > 0, "Word count must be positive"

Snapshot testing for prompt regression

When you change a prompt, snapshot testing captures the output shape and fails on unexpected structural changes:

import json
from pathlib import Path

SNAPSHOT_DIR = Path("tests/snapshots")


def load_snapshot(name: str) -> dict | None:
    path = SNAPSHOT_DIR / f"{name}.json"
    return json.loads(path.read_text()) if path.exists() else None


def save_snapshot(name: str, data: dict) -> None:
    SNAPSHOT_DIR.mkdir(exist_ok=True)
    (SNAPSHOT_DIR / f"{name}.json").write_text(json.dumps(data, indent=2))


def assert_snapshot_keys(name: str, actual: dict, update: bool = False) -> None:
    if update:
        save_snapshot(name, actual)
        return
    expected = load_snapshot(name)
    if expected is None:
        save_snapshot(name, actual)
        return
    assert set(actual.keys()) == set(expected.keys()), (
        f"Snapshot key mismatch for {name}: got {set(actual.keys())}, expected {set(expected.keys())}"
    )

Run with update=True when you intentionally change a prompt. CI runs without update catch accidental structural regressions.

Eval-driven testing with golden datasets

For quality gates, run real LLM calls against a curated dataset with known good properties:

import asyncio
import json
from pathlib import Path
from dataclasses import dataclass, field


@dataclass
class EvalCase:
    id: str
    input: str
    expected_entities: list[str]
    expected_sentiment: str
    min_confidence: float = 0.7


@dataclass
class EvalResult:
    case_id: str
    passed: bool
    errors: list[str] = field(default_factory=list)


async def run_eval_suite(extract_fn, dataset_path: Path) -> tuple[list[EvalResult], float]:
    cases = [EvalCase(**c) for c in json.loads(dataset_path.read_text())]
    results = []

    for case in cases:
        result = await extract_fn(case.input)
        errors = []

        if result is None:
            errors.append("extraction returned None")
        else:
            missing = [e for e in case.expected_entities if e not in result.entities]
            if missing:
                errors.append(f"missing entities: {missing}")
            if result.sentiment != case.expected_sentiment:
                errors.append(f"sentiment mismatch: expected {case.expected_sentiment!r}, got {result.sentiment!r}")
            if result.confidence < case.min_confidence:
                errors.append(f"confidence {result.confidence:.2f} below threshold {case.min_confidence}")

        results.append(EvalResult(case_id=case.id, passed=len(errors) == 0, errors=errors))

    pass_rate = sum(1 for r in results if r.passed) / len(results)
    return results, pass_rate

Common mistakes

Asserting exact LLM output strings. These tests break on model upgrades, temperature variation, and prompt tweaks. Assert structure and properties instead.

Testing against live APIs in every test run. Unit tests should not hit provider APIs. Mock at the client level and reserve real API calls for scheduled eval runs.

No golden dataset maintenance. Golden datasets rot. When you improve the system, update the expected outputs. When you add a capability, add new eval cases.

Configuring the mock to return an unconfigured MagicMock. An unconfigured MagicMock() will not surface shape errors in your parsing code. Configure the mock return value to match the actual SDK response shape.

Try it yourself

1. Write a pytest fixture that mocks an LLM client. Write three tests that use it with different response configurations — one valid, one with a missing field, one returning None.
2. Add structural assertions to an extraction function you have. Verify both a valid mock response and a response with a missing required field are detected correctly.
3. Create a golden dataset with 5 test cases for a feature you are building. Write an eval runner that loads the dataset, calls your function, and reports a pass rate with per-case failure details.