AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Build a RAG evaluation harness with Recall@K and MRR

Build a RAG Evaluation Harness with Recall@K and MRR

You cannot improve what you cannot measure. Most RAG pipelines are deployed and then tuned based on user complaints — the worst possible feedback loop. A proper evaluation harness gives you Recall@K and MRR measurements before deploying a change, so you know whether a new chunking strategy, embedding model, or retrieval configuration actually improved anything.

What you are building

Create a RAGEvalHarness class that:

Accepts a test set — a list of TestCase objects, each with a query and relevant_doc_ids (IDs of documents that should be retrieved).
Accepts an injectable retrieval function — Callable[[str, int], list[str]] returning document IDs given query and k.
Computes Recall@K for K values 1, 3, 5, and 10 — fraction of test cases where at least one relevant document appears in the top K.
Computes Mean Reciprocal Rank (MRR) — mean of 1/rank of the first relevant result (0 if not found).
Returns an EvalReport with per-query results, aggregate metrics, and failure cases.
Identifies failure cases — queries where no relevant document appears in the top 10.

Why this matters

Recall@K tells you whether the right document is even in the retrieved set. MRR tells you whether it is near the top. Without these numbers, retrieval changes are guesswork. A 10-minute eval harness saves weeks of debugging in production.

Constraints

Standard library only. Injectable retrieve function. Type-hint everything.

Retrieval / medium / Step 11 of 15

Practice stage

Retrieval quality and ranking

Hint

Do not trust a single score blindly. Combine ranking logic with metadata and think about why a candidate deserves to rise.

Next drill

Implement query expansion for improved RAG recall

Success criteria

- Ranking rule is explainable
- Supports future iteration without rewriting everything
- Feels tied to product trust rather than pure math trivia

Review checklist

- Did I account for both relevance and metadata?
- Would I be able to explain the ranking rule in an interview?
- Does the code make future tuning easy?

from __future__ import annotations

from dataclasses import dataclass, field
from typing import Callable

@dataclass
class TestCase:
    query: str
    relevant_doc_ids: list[str]
    description: str = ""

@dataclass
class QueryResult:
    query: str
    relevant_doc_ids: list[str]
    retrieved_ids: list[str]
    recall_at_k: dict[int, bool]
    reciprocal_rank: float

@dataclass
class EvalReport:
    recall_at_1: float
    recall_at_3: float
    recall_at_5: float
    recall_at_10: float
    mrr: float
    n_queries: int
    failure_cases: list[QueryResult]
    per_query: list[QueryResult]

class RAGEvalHarness:
    def __init__(
        self,
        retrieve_fn: Callable[[str, int], list[str]],
        k_values: list[int] | None = None,
    ):
        self.retrieve_fn = retrieve_fn
        self.k_values = k_values or [1, 3, 5, 10]

def _compute_reciprocal_rank(
        self, retrieved_ids: list[str], relevant_ids: list[str]
    ) -> float:
        """Return 1/rank of the first relevant document, 0 if not found."""
        # TODO: implement
        raise NotImplementedError

def evaluate(self, test_cases: list[TestCase]) -> EvalReport:
        """Run evaluation and return aggregate report with failure cases."""
        # TODO: implement
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.