AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Evaluate RAG faithfulness with citation tracking

Evaluate RAG Faithfulness with Citation Tracking

A RAG answer is only as trustworthy as its grounding. Faithfulness evaluation measures whether each claim in the answer is supported by the retrieved context. Citation tracking extends this: can you point to the exact chunk that each claim came from?

What you are building

Create a FaithfulnessEvaluator that:

Extracts claims from a generated answer (one per sentence).
Matches each claim to the most relevant retrieved chunk using cosine similarity.
Classifies each claim as SUPPORTED (similarity above threshold) or UNSUPPORTED.
Computes a faithfulness score as the fraction of supported claims.
Builds a citation map linking each supported claim to its best-matching chunk ID.
Generates a human-readable summary of the evaluation result.

Why this matters

This is the core evaluation loop in every production RAG system that cares about auditability. Building it yourself means you understand what faithfulness actually measures, how to tune thresholds, and how to debug low scores.

Constraints

Use cosine similarity for matching (inject the embedding function — no LLM judge).
No external evaluation libraries.
Return an EvaluationResult dataclass with all detail needed to debug a failing case.
Default similarity threshold: 0.75 (configurable).

Rag Systems / medium / Step 4 of 6

Practice stage

General drill

Hint

Keep the solution explicit and reviewable.

Next drill

Build an incremental document indexer

Success criteria

Make the solution explicit, debuggable, and easy to explain.

Review checklist

Review where the boundary is, what gets validated, and what would be hard to debug later.

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import Callable

@dataclass
class RetrievedChunk:
    id: str
    text: str
    metadata: dict = field(default_factory=dict)

@dataclass
class ClaimVerdict:
    claim: str
    verdict: str           # "SUPPORTED" or "UNSUPPORTED"
    best_chunk_id: str
    similarity: float

@dataclass
class EvaluationResult:
    question: str
    answer: str
    verdicts: list[ClaimVerdict]
    faithfulness_score: float
    citation_map: dict[str, str]   # claim -> chunk_id
    summary: str

class FaithfulnessEvaluator:
    def __init__(
        self,
        embed_fn: Callable[[str], list[float]],
        similarity_threshold: float = 0.75,
    ):
        self.embed_fn = embed_fn
        self.threshold = similarity_threshold

def _extract_claims(self, text: str) -> list[str]:
        # TODO: split on sentence boundaries, filter short fragments
        raise NotImplementedError

def _cosine(self, a: list[float], b: list[float]) -> float:
        # TODO: implement
        raise NotImplementedError

def evaluate(
        self,
        question: str,
        answer: str,
        retrieved_chunks: list[RetrievedChunk],
    ) -> EvaluationResult:
        # TODO: extract claims, embed, match to chunks, classify, score
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.