RAG evaluation and debugging

Why this matters

A RAG system fails in three distinct ways that require three distinct fixes: wrong documents were retrieved, the right documents were retrieved but the model ignored them, or the model answered correctly but cited the wrong sources. If you do not measure these layers separately, you will spend weeks prompt-engineering when the real problem is chunking, or rewriting ingestion pipelines when the real problem is the prompt.

This lesson gives you the evaluation framework and debugging workflow to tell these failures apart before you spend a day investigating the wrong thing.

Core concepts

Faithfulness. Does the model's answer follow from the retrieved context? A faithful answer makes only claims that are supported by the retrieved chunks. An unfaithful answer adds information from the model's parametric knowledge — which might be correct, but is not grounded and cannot be cited. Measure faithfulness with an LLM judge that checks each claim in the answer against the provided chunks.

Context relevance. Are the retrieved chunks actually relevant to the question? You can have a perfectly faithful answer but a terrible RAG system if the chunks are irrelevant — the model is just doing its best with garbage input. Measure context relevance as the fraction of retrieved chunks that contain information useful for answering the query.

Answer relevance. Does the final answer address what the user actually asked? A system can be faithful and context-relevant but still produce an answer that talks around the question without directly addressing it.

The evaluation-retrieval-generation framework. When debugging, always isolate the layer that is failing:

1. Sample 20 queries from real or realistic usage.
2. For each query, retrieve and log the top-k chunks.
3. Ask: does any chunk contain the correct answer? (Retrieval quality check.)
4. If yes, ask: does the model use that information in its answer? (Generation quality check.)
5. If the chunk is not retrieved, ask: is the document in the index? (Ingestion check.) Was it chunked in a way that separates the relevant information? (Chunking check.)

RAGAS. The open-source RAGAS library automates faithfulness, answer relevance, and context precision/recall measurement using an LLM-as-judge approach. Useful for batch evaluation over a test set.

Working example

from __future__ import annotations
import json
from openai import OpenAI

client = OpenAI()


def evaluate_faithfulness(
    question: str,
    answer: str,
    retrieved_chunks: list[str],
    model: str = "gpt-4o-mini",
) -> dict:
    """
    LLM-as-judge: check if each claim in the answer is supported
    by the retrieved chunks. Returns a score 0-1 and a reasoning trace.
    """
    context = "\n\n---\n\n".join(
        f"[Chunk {i+1}]: {chunk}" for i, chunk in enumerate(retrieved_chunks)
    )

    prompt = f"""You are evaluating a RAG system answer for faithfulness.

Question: {question}

Retrieved Context:
{context}

Answer to evaluate:
{answer}

Task: Identify each factual claim in the answer. For each claim, determine whether it is:
- SUPPORTED: Directly stated or clearly implied by the retrieved context
- UNSUPPORTED: Not present in the context (may be from model knowledge or hallucinated)

Return a JSON object with:
- "claims": list of {{claim: str, verdict: "SUPPORTED" | "UNSUPPORTED", evidence: str}}
- "faithfulness_score": fraction of claims that are SUPPORTED (0.0 to 1.0)
- "summary": one sentence explaining the result

Respond with only the JSON object."""

    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )

    return json.loads(response.choices[0].message.content)


def evaluate_context_relevance(
    question: str,
    retrieved_chunks: list[str],
    model: str = "gpt-4o-mini",
) -> dict:
    """
    For each retrieved chunk, score whether it is relevant to the question.
    Returns per-chunk verdicts and an overall precision score.
    """
    results = []
    for i, chunk in enumerate(retrieved_chunks):
        prompt = f"""Is the following text relevant to answering this question?

Question: {question}

Text: {chunk}

Respond with JSON: {{"relevant": true/false, "reason": "brief explanation"}}"""

        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            response_format={"type": "json_object"},
        )
        verdict = json.loads(response.choices[0].message.content)
        results.append({"chunk_index": i, **verdict})

    precision = sum(1 for r in results if r["relevant"]) / len(results) if results else 0.0
    return {"chunks": results, "context_precision": precision}


def run_eval_suite(
    test_cases: list[dict],
    rag_pipeline,  # callable: query -> (answer, retrieved_chunks)
) -> dict:
    """
    Run a batch evaluation over a test set.
    Each test case: {"question": str, "expected_answer": str (optional)}
    """
    results = []
    for case in test_cases:
        question = case["question"]
        answer, chunks = rag_pipeline(question)

        faithfulness = evaluate_faithfulness(question, answer, chunks)
        relevance = evaluate_context_relevance(question, chunks)

        results.append({
            "question": question,
            "answer": answer,
            "faithfulness_score": faithfulness["faithfulness_score"],
            "context_precision": relevance["context_precision"],
            "unsupported_claims": [
                c for c in faithfulness["claims"]
                if c["verdict"] == "UNSUPPORTED"
            ],
        })

    avg_faithfulness = sum(r["faithfulness_score"] for r in results) / len(results)
    avg_precision = sum(r["context_precision"] for r in results) / len(results)

    return {
        "results": results,
        "summary": {
            "n": len(results),
            "avg_faithfulness": round(avg_faithfulness, 3),
            "avg_context_precision": round(avg_precision, 3),
        },
    }


# --- Debugging workflow ---
def debug_retrieval_failure(
    question: str,
    index,  # Your vector index
    expected_source: str,
    top_k: int = 10,
) -> None:
    """
    When a known question is not being answered correctly,
    diagnose whether it is a retrieval problem or a generation problem.
    """
    print(f"Query: {question}")
    print(f"Expected source: {expected_source}")
    print()

    chunks = index.search(question, top_k=top_k)

    found_expected = False
    for i, chunk in enumerate(chunks):
        source = chunk.metadata.get("source", "unknown")
        is_expected = expected_source in source
        marker = "<<< TARGET" if is_expected else ""
        found_expected = found_expected or is_expected
        print(f"  [{i+1}] dist={chunk.distance:.3f} source={source} {marker}")
        print(f"       {chunk.text[:100]}...")
        print()

    if not found_expected:
        print("DIAGNOSIS: Target document not in top results.")
        print("  -> Check: Is the document in the index?")
        print("  -> Check: Are chunks too large (semantic dilution)?")
        print("  -> Check: Does the query use different terminology than the document?")
        print("  -> Try: Hybrid search to catch exact term matches")
    else:
        print("DIAGNOSIS: Target document WAS retrieved.")
        print("  -> Problem is likely in generation (model not using the context)")
        print("  -> Check: Is the relevant chunk in the top-3 or only in 7-10?")
        print("  -> Try: Move relevant chunk higher via reranking")
        print("  -> Try: Check if your prompt instructs the model to use context")

Common mistakes

1. Evaluating only the final answer. If the answer is wrong, you do not know if retrieval failed or generation failed without inspecting retrieved chunks. Always log what the model received.

2. Using the same LLM that generates answers to judge them. A judge model should ideally be different from (or more capable than) the generation model. Otherwise you are asking the model whether its own output is good.

3. Measuring faithfulness without measuring context relevance. A system can score 100% on faithfulness (all claims are in context) while having terrible context relevance (the context is irrelevant and the model says "I don't know"). Both metrics together tell the full story.

4. Building an eval suite with only easy queries. Test with ambiguous queries, queries that span multiple documents, and queries that have no answer in the corpus. The hard cases reveal where the system breaks down.

5. Not tracking evaluation scores over time. Run your eval suite on every significant change to chunking, embedding model, or retrieval parameters. Without regression tracking, improvements and regressions are invisible.

Try it yourself

Take your RAG pipeline (or build a minimal one) and create a test set of 10 questions where you know what the correct answer source is. Run the debug_retrieval_failure function for any question where the answer is wrong. Determine for each failure whether it is a retrieval problem (right chunk not retrieved) or a generation problem (right chunk retrieved but not used). This diagnostic tells you whether to invest in retrieval improvements or prompt improvements.