AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Agent evaluation harness

Agent Evaluation Harness

Building an agent is one thing; knowing whether it actually works is another. An evaluation harness runs an agent against a suite of test cases and scores it on multiple dimensions: correctness (did it get the right answer?), efficiency (how many steps did it take?), and cost (how many tokens did it use?).

What you are building

Create an EvalHarness class that:

Defines test cases — each case has an input, expected output, and optional metadata (max steps, category).
Runs the agent — executes the agent function for each test case, capturing the result, step trace, and token usage.
Scores correctness — uses a configurable scoring function (exact match, fuzzy match, or LLM-as-judge).
Scores efficiency — compares actual steps taken against a baseline or maximum.
Scores cost — tracks total tokens and computes cost based on a pricing table.
Produces a report — aggregate scores by category, overall pass rate, and per-case details.

Why this matters

Without systematic evaluation, agent development is guesswork. You change a prompt and hope it helps. An eval harness turns agent development into engineering: make a change, run the suite, see exactly what improved and what regressed. This is the single most important infrastructure investment in any agent project.

Every serious AI engineering team has an eval harness. The patterns here apply whether you are using a framework or building from scratch.

Constraints

Test cases are defined as dicts/dataclasses, not hardcoded.
The harness must be agent-agnostic — it accepts any callable with the right signature.
Scoring functions are pluggable (passed in, not hardcoded).
The report must include both aggregate and per-case results.
Support timeout per test case.

Agents / advanced / Step 7 of 8

Practice stage

Agent architecture patterns

Hint

Focus on explicit control surfaces — function schemas, state machines, and structured outputs. Agents are most useful when the task requires dynamic tool selection or multi-step reasoning. Start with the simplest pattern (single tool call) before reaching for ReAct loops.

Next drill

Cost-tracking middleware for agent tool calls

Success criteria

- Tools return structured, typed responses
- Agent completes the task within a bounded number of steps
- All tool calls include error handling and retries
- Memory/state management prevents unbounded context growth

Review checklist

- Tool schemas validate inputs and handle errors gracefully
- Agent loop has explicit termination conditions
- State is serializable and inspectable between steps
- Cost and token usage are tracked per invocation
- Fallback behavior exists for tool call failures

from __future__ import annotations

import time
from dataclasses import dataclass, field
from typing import Any, Callable

@dataclass
class TestCase:
    """A single evaluation test case."""
    case_id: str
    input_data: Any
    expected_output: Any
    max_steps: int | None = None
    category: str = "default"
    metadata: dict[str, Any] = field(default_factory=dict)

@dataclass
class CaseResult:
    """Result from running one test case."""
    case_id: str
    passed: bool
    correctness_score: float  # 0.0 to 1.0
    actual_output: Any = None
    steps_taken: int = 0
    tokens_used: int = 0
    latency_ms: float = 0.0
    error: str | None = None

@dataclass
class EvalReport:
    """Aggregate evaluation report."""
    total_cases: int
    passed: int
    failed: int
    overall_pass_rate: float
    avg_correctness: float
    avg_steps: float
    total_tokens: int
    total_cost_usd: float
    by_category: dict[str, dict] = field(default_factory=dict)
    case_results: list[CaseResult] = field(default_factory=list)

@dataclass
class AgentOutput:
    """Standard output format from an agent under evaluation."""
    result: Any
    steps: int = 0
    tokens: int = 0

class EvalHarness:
    """Runs an agent against test cases and produces a scored report."""

def __init__(
        self,
        agent_fn: Callable[[Any], AgentOutput],
        scorer: Callable[[Any, Any], float],
        pricing: dict[str, float] | None = None,
        timeout: float = 30.0,
    ) -> None:
        self.agent_fn = agent_fn
        self.scorer = scorer
        self.pricing = pricing or {"input": 0.01, "output": 0.03}  # per 1K tokens
        self.timeout = timeout

def run_case(self, case: TestCase) -> CaseResult:
        """Run a single test case and score it."""
        # TODO: call agent_fn with case.input_data
        # TODO: measure latency
        # TODO: score correctness using self.scorer
        # TODO: handle timeout and exceptions
        raise NotImplementedError

def run_suite(self, cases: list[TestCase]) -> EvalReport:
        """Run all test cases and produce an aggregate report."""
        # TODO: run each case
        # TODO: aggregate scores by category
        # TODO: compute overall metrics
        # TODO: calculate cost from token usage
        raise NotImplementedError

def _compute_cost(self, tokens: int) -> float:
        """Compute cost from token usage."""
        # TODO: apply pricing
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.