Your personal operating system for career transition.

Measuring agent effectiveness

Most agent projects fail not because the agent produces wrong answers, but because the team cannot tell whether it is working.

"It seemed fine in testing" is not a measurement. You need metrics that tell you what the agent is doing, how much it costs, where it fails, and whether a change made things better or worse.

This article covers five evaluation layers with practical implementation guidance.

---

Layer 1: Task completion rate

Task completion rate is the fraction of tasks the agent completes successfully versus fails, times out, or produces an unacceptable result.

from dataclasses import dataclass
from enum import Enum

class TaskOutcome(Enum):
    SUCCESS = "success"
    PARTIAL = "partial"          # Completed but with degraded quality
    TOOL_FAILURE = "tool_failure"
    LOOP_LIMIT = "loop_limit"    # Hit max iterations without finishing
    TIMEOUT = "timeout"
    MODEL_ERROR = "model_error"

@dataclass
class TaskTrace:
    task_id: str
    input: str
    outcome: TaskOutcome
    steps_taken: int
    final_output: str | None
    error: str | None
    duration_ms: int
    total_tokens: int

How to define success: Write a success definition before you start measuring. "The agent answered the question" is not a definition. "The agent returned a structured response that includes a valid invoice ID and a non-zero amount" is.

def evaluate_invoice_task(trace: TaskTrace, expected: dict) -> bool:
    if trace.outcome != TaskOutcome.SUCCESS:
        return False
    output = parse_agent_output(trace.final_output)
    return (
        output.get("invoice_id", "").startswith("inv_") and
        output.get("amount", 0) > 0 and
        output.get("status") in ("paid", "pending", "overdue")
    )

Useful breakdowns:

• Completion rate by task category
• Completion rate by task complexity (steps required)
• Completion rate over time (catch regressions after deploys)

A task completion rate below 85% on your target task type usually means either tool reliability problems or task definition is too open-ended for the current agent design.

---

Layer 2: Cost efficiency (tokens per task)

Total tokens per completed task is the single most useful cost metric. It combines prompt tokens, completion tokens, and retries in one number.

def compute_cost_metrics(traces: list[TaskTrace]) -> dict:
    successful = [t for t in traces if t.outcome == TaskOutcome.SUCCESS]
    failed = [t for t in traces if t.outcome != TaskOutcome.SUCCESS]

    return {
        "total_tasks": len(traces),
        "completion_rate": len(successful) / len(traces) if traces else 0,
        "avg_tokens_per_completed_task": (
            sum(t.total_tokens for t in successful) / len(successful)
            if successful else 0
        ),
        "avg_tokens_per_failed_task": (
            sum(t.total_tokens for t in failed) / len(failed)
            if failed else 0
        ),
        "avg_steps_per_completed_task": (
            sum(t.steps_taken for t in successful) / len(successful)
            if successful else 0
        ),
    }

Cost efficiency benchmarks to track:

• Tokens per completed task (lower is better, track trend not absolute)
• Tokens burned on failed tasks (pure waste — reduce failures to reduce this)
• Steps per task (a proxy for latency and model call count)

Practical targets: If average tokens per task are growing week-over-week without a corresponding improvement in quality, the agent is becoming less efficient. Common causes: longer prompts from accumulated system prompt edits, more tool calls per task due to retrieval degradation, or increased retry rates.

---

Layer 3: Reliability (retry and failure rates)

Reliability measures how often the agent needs to retry a tool call and how often retries help versus hurt.

@dataclass
class ToolCallRecord:
    tool_name: str
    attempt_number: int    # 1-indexed; 2+ means retry
    success: bool
    error_code: str | None
    duration_ms: int
    input_hash: str        # Hash of inputs, for detecting identical retries

def compute_reliability_metrics(tool_calls: list[ToolCallRecord]) -> dict:
    by_tool = {}
    for call in tool_calls:
        name = call.tool_name
        if name not in by_tool:
            by_tool[name] = {"total": 0, "retries": 0, "failures": 0}
        by_tool[name]["total"] += 1
        if call.attempt_number > 1:
            by_tool[name]["retries"] += 1
        if not call.success:
            by_tool[name]["failures"] += 1

    return {
        tool: {
            "retry_rate": stats["retries"] / stats["total"],
            "failure_rate": stats["failures"] / stats["total"],
        }
        for tool, stats in by_tool.items()
    }

Thresholds to watch:

• Retry rate > 10% on any tool — The tool schema is probably too loose, or the tool is fragile. Investigate.
• Failure rate > 5% — The tool has a reliability problem. Check error codes to determine whether it is invalid inputs (schema issue), upstream failures (external service), or logic errors.
• Identical retries — Same input hash retried twice means the agent is not learning from the first failure. Add error context to the observation before retrying.

---

Layer 4: A/B testing agents

A/B testing compares two agent configurations against the same task set and picks the winner on your chosen metric.

import random

def run_ab_test(
    tasks: list[dict],
    agent_a,
    agent_b,
    split: float = 0.5,
    metric_fn = None
) -> dict:
    results_a, results_b = [], []

    for task in tasks:
        if random.random() < split:
            trace = agent_a.run(task)
            results_a.append(trace)
        else:
            trace = agent_b.run(task)
            results_b.append(trace)

    metrics_a = compute_cost_metrics(results_a)
    metrics_b = compute_cost_metrics(results_b)

    return {
        "agent_a": metrics_a,
        "agent_b": metrics_b,
        "winner": "a" if metrics_a["completion_rate"] > metrics_b["completion_rate"] else "b",
        "delta_completion_rate": metrics_a["completion_rate"] - metrics_b["completion_rate"],
        "delta_avg_tokens": metrics_a["avg_tokens_per_completed_task"] - metrics_b["avg_tokens_per_completed_task"],
    }

A/B testing discipline:

• Use the same task set for both agents. Random assignment is fine if the set is large enough.
• Run at minimum 50 tasks per arm before drawing conclusions. Smaller samples produce noise, not signal.
• Define your primary metric before running the test. Completion rate is usually the primary metric; token cost is secondary. If you optimize for cost first, you often degrade quality.
• Test one change at a time. Model swap, prompt change, and tool schema change all in one A/B is impossible to interpret.

---

Layer 5: Offline eval harnesses

An offline eval harness runs your agent against a fixed task dataset on demand, without production traffic. It is the foundation of regression testing.

# eval/run_harness.py
import json
from pathlib import Path
from app.agent import build_agent

EVAL_TASKS_PATH = Path("eval/tasks/invoice_tasks.jsonl")

def load_tasks() -> list[dict]:
    with EVAL_TASKS_PATH.open() as f:
        return [json.loads(line) for line in f]

def run_harness(agent_config: dict) -> dict:
    agent = build_agent(**agent_config)
    tasks = load_tasks()
    traces = []

    for task in tasks:
        trace = agent.run(task["input"])
        success = evaluate_task(trace, task["expected"])
        traces.append({
            "task_id": task["id"],
            "success": success,
            "tokens": trace.total_tokens,
            "steps": trace.steps_taken,
            "outcome": trace.outcome.value,
        })

    completion_rate = sum(1 for t in traces if t["success"]) / len(traces)
    return {
        "completion_rate": completion_rate,
        "avg_tokens": sum(t["tokens"] for t in traces) / len(traces),
        "traces": traces,
    }

if __name__ == "__main__":
    results = run_harness({"model": "claude-3-5-sonnet-20241022", "max_steps": 10})
    print(f"Completion rate: {results['completion_rate']:.1%}")
    print(f"Avg tokens per task: {results['avg_tokens']:.0f}")

Eval dataset rules:

• Start with 20–30 hand-curated tasks. Larger is not better if the tasks are vague. Quality over quantity.
• Cover failure modes explicitly. Include tasks with missing data, ambiguous inputs, and edge cases — not just happy paths.
• Never add tasks that were used to tune the prompt. This inflates your eval score. Keep tuning tasks and eval tasks separate.
• Run the harness before every significant change. Model swap, prompt edit, tool schema change — run it every time.

---

Putting it together: a weekly eval ritual

A practical weekly cadence for a production agent team:

1. Check dashboards (5 min): Completion rate, avg tokens, retry rates — any week-over-week change above 5% triggers investigation.
2. Run offline harness (automated, on every PR): Gate deploys on completion rate not dropping more than 2 percentage points.
3. Sample 10 failed traces (15 min): Read the full trace, identify failure category (tool failure, model error, bad schema, loop limit). Add one new eval task if you found a new failure pattern.
4. Check cost trend (5 min): Tokens per task trending up? Find the source before it doubles.

---

The question to keep asking

The right question is not "did the agent succeed?" The right question is "do I understand why it succeeds when it does, and why it fails when it does?"

You have that understanding when you can point to a metric, a trace, and a cause for every significant change in agent behavior. Until then, keep adding instrumentation.