Why this matters
Eval frameworks like promptfoo and RAGAS are useful, but every serious AI team eventually hits something the framework cannot do: a custom scoring rubric, a domain-specific assertion, a reporter format their dashboards expect, or a latency constraint the framework ignores. Knowing how to build a harness from scratch means you are never blocked by your tooling. It also means you deeply understand what frameworks are doing under the hood, so you use them better when you do reach for one.
Core concepts
When to use a framework vs. a custom harness
Use a framework when: you are prototyping, the built-in scorers cover your use case, and your team does not need to own the eval infrastructure.
Use a custom harness when: you have scoring logic that no framework supports, you need the harness to run inside your existing CI pipeline without additional dependencies, or you want full control over how results are stored and surfaced.
The pragmatic answer: start with a framework, extract what you need into your own code when the framework becomes friction.
Harness architecture
A minimal harness has four components:
- Test cases — inputs plus expected outputs or acceptance criteria
- Runner — calls the system under test for each case
- Scorers — evaluate the output against the criteria
- Reporter — formats and persists results
from dataclasses import dataclass, field
from typing import Any, Callable
import json
import time
@dataclass
class TestCase:
id: str
input: dict[str, Any]
expected: Any = None
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass
class EvalResult:
case_id: str
output: Any
scores: dict[str, float]
latency_ms: float
passed: bool
error: str | None = None
Scorer = Callable[[TestCase, Any], float]
class EvalHarness:
def __init__(self, system_fn: Callable, scorers: dict[str, Scorer]):
self.system_fn = system_fn
self.scorers = scorers
def run(self, cases: list[TestCase]) -> list[EvalResult]:
results = []
for case in cases:
start = time.perf_counter()
try:
output = self.system_fn(**case.input)
latency_ms = (time.perf_counter() - start) * 1000
scores = {
name: scorer(case, output)
for name, scorer in self.scorers.items()
}
passed = all(s >= 0.5 for s in scores.values())
results.append(EvalResult(case.id, output, scores, latency_ms, passed))
except Exception as exc:
latency_ms = (time.perf_counter() - start) * 1000
results.append(EvalResult(case.id, None, {}, latency_ms, False, str(exc)))
return results
def report_json(self, results: list[EvalResult], path: str) -> dict:
summary = {
"total": len(results),
"passed": sum(1 for r in results if r.passed),
"failed": sum(1 for r in results if not r.passed),
"avg_latency_ms": sum(r.latency_ms for r in results) / len(results),
"results": [vars(r) for r in results],
}
with open(path, "w") as f:
json.dump(summary, f, indent=2)
return summary
Deterministic vs LLM-as-judge vs human scorers
Deterministic — exact match, substring, regex, JSON schema validation. Fast, free, fully repeatable. Use whenever the answer space is constrained.
def exact_match_scorer(case: TestCase, output: Any) -> float:
return 1.0 if output == case.expected else 0.0
def json_valid_scorer(case: TestCase, output: Any) -> float:
try:
json.loads(output) if isinstance(output, str) else output
return 1.0
except Exception:
return 0.0
LLM-as-judge — use a separate LLM call to score freeform outputs. Flexible but adds cost and non-determinism. Use for tone, helpfulness, coherence, safety.
import anthropic
def llm_judge_scorer(rubric: str) -> Scorer:
client = anthropic.Anthropic()
def scorer(case: TestCase, output: Any) -> float:
prompt = f"{rubric}
Input: {case.input}
Output: {output}
Score 0.0 to 1.0 only, no explanation."
response = client.messages.create(
model="claude-3-5-haiku-20241022",
max_tokens=10,
messages=[{"role": "user", "content": prompt}],
)
try:
return float(response.content[0].text.strip())
except ValueError:
return 0.0
return scorer
Human scorers — write results to a CSV or annotation UI for human review. Required for high-stakes decisions where LLM judges are not trustworthy enough.
Putting it together
test_cases = [
TestCase("q1", {"question": "What is 2+2?"}, expected="4"),
TestCase("q2", {"question": "Summarize this: ..."}, expected=None),
]
harness = EvalHarness(
system_fn=my_qa_function,
scorers={
"exact": exact_match_scorer,
"json_valid": json_valid_scorer,
},
)
results = harness.run(test_cases)
summary = harness.report_json(results, "eval_results.json")
print(f"Passed {summary['passed']}/{summary['total']} — avg latency {summary['avg_latency_ms']:.0f}ms")
Common mistakes
Coupling scorer logic to the runner — scorers should be pure functions that take a case and an output. If a scorer reads from a database or calls an external API as a side effect, it becomes impossible to test in isolation.
No error isolation — if one case crashes the runner, you lose results for all subsequent cases. Always catch exceptions per case and record them as failures.
Averaging scores naively — a single low score on a critical safety case should not be averaged away by dozens of high scores on easy cases. Design your pass/fail thresholds case-by-case or category-by-category.
LLM judge prompt drift — the rubric given to an LLM judge is itself a prompt that can be changed. Version control your judge prompts just like production prompts.
Try it yourself
Build a working eval harness for a summarization function of your choice. Requirements: at least three test cases, at least two different scorer types (one deterministic, one LLM-as-judge), JSON output written to disk, and a printed summary showing pass rate and average latency. Then add a fourth scorer that checks whether the summary is shorter than the input — implement it as a deterministic scorer that returns 1.0 if the output word count is less than the input word count, 0.0 otherwise.