Why this matters

Naive RAG — embed query, retrieve top-k chunks, generate answer — works well for simple factual lookups. It fails on questions that require connecting information across multiple documents, questions where the first retrieval misses the relevant content, and questions where the retrieved chunks are confidently wrong. These are the exact cases that matter most in production: the complex, ambiguous, hard questions that users ask when they actually need the system.

Advanced RAG patterns address three failure modes: insufficient reasoning depth (multi-hop RAG), overconfident generation on weak retrieval (self-RAG), and silent retrieval failure (corrective RAG). Understanding these patterns gives you the vocabulary and implementation tools to diagnose which failure mode your system has and apply the right fix.

Core concepts

Multi-hop RAG

Multi-hop RAG handles questions that require combining information from multiple independent sources. The pattern: decompose the original question into sub-questions, retrieve independently for each sub-question, synthesize the intermediate answers into a final response.

Example question: "Who founded the company that acquired DeepMind, and what was their original product?" This requires three retrieval steps: (1) who acquired DeepMind, (2) who founded that company, (3) what was their original product. A single retrieval pass will not surface all three pieces in one chunk.

Query decomposition is the first step. You can use an LLM to generate sub-questions, or use explicit decomposition prompts. The key is making sub-questions independently answerable and covering the full information need.

Sequential vs parallel retrieval. Some sub-questions depend on the answer to a previous one ("who founded [company X]" requires knowing company X first). Others are independent and can be retrieved in parallel. Identify dependencies before deciding the retrieval order.

Synthesis. After collecting intermediate answers, use a final generation call with all intermediate answers in context to produce the combined response. The synthesis prompt should explicitly reference the intermediate answers.

Self-RAG

Self-RAG adds a self-checking loop: after retrieving and generating, the model evaluates its own output for factual support. If the generated answer is not well-supported by the retrieved chunks, the system re-retrieves with a refined query and regenerates.

The core insight: a generation that contradicts the retrieved context, or a context that the model is uncertain about, should trigger re-retrieval rather than being surfaced to the user. This is a quality gate inside the RAG loop.

Confidence signal. You can implement confidence checking via a separate LLM call ("Does this answer follow directly from the context? Reply yes or no."), via log-probability analysis of the generated tokens, or via structured self-evaluation prompts that ask the model to rate its own confidence.

Re-retrieval strategy. When confidence is low, generate a refined query based on what was missing. If the first query was "Python asyncio event loop" and the retrieved context was insufficient, the refined query might be "how Python asyncio event loop executes coroutines internally". Query refinement is the mechanism that makes re-retrieval more likely to succeed.

Loop limit. Always cap the number of re-retrieval iterations. One or two retries is usually sufficient. More than that and you are likely dealing with a knowledge gap that no amount of re-retrieval will fix.

Corrective RAG

Corrective RAG detects low-confidence retrieval early — before generation — and falls back to alternative retrieval strategies. The pattern: retrieve → score retrieval quality → if quality is low, apply fallback (web search, different index, query expansion) → then generate.

Retrieval quality signals: average cosine similarity of top-k results (low similarity means the question is far from any indexed content), number of results below a similarity threshold, the spread of similarity scores (a large gap between result #1 and result #5 suggests sparse coverage).

Fallback strategies: (1) web search — integrate a search API to retrieve live content when the index lacks the answer, (2) query expansion — generate query variants and re-retrieve, (3) broader scope — relax metadata filters and retrieve from a larger document set, (4) explicit "I don't know" — for very low confidence, return a structured no-answer rather than hallucinating.

Working example

from __future__ import annotations
from dataclasses import dataclass

from openai import OpenAI

client = OpenAI()


@dataclass
class RetrievalResult:
    text: str
    score: float
    metadata: dict


@dataclass
class SelfRAGResult:
    answer: str
    iterations: int
    final_confidence: str  # "high" | "low"
    queries_used: list[str]


def embed_query(query: str) -> list[float]:
    resp = client.embeddings.create(model="text-embedding-3-small", input=query)
    return resp.data[0].embedding


def retrieve(query: str, index, top_k: int = 5) -> list[RetrievalResult]:
    """Retrieve top-k results from your vector index."""
    embedding = embed_query(query)
    raw = index.query(embedding, top_k=top_k)
    return [RetrievalResult(text=r["text"], score=r["score"], metadata=r["metadata"]) for r in raw]


def generate_answer(query: str, context: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer the question using only the provided context. Be precise."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
        ],
        temperature=0.1,
        max_tokens=512,
    )
    return resp.choices[0].message.content


def check_answer_confidence(query: str, context: str, answer: str) -> str:
    """Ask the model to self-evaluate whether the answer is well-supported. Returns 'high' or 'low'."""
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a factual accuracy evaluator. Given a question, retrieved context, "
                    "and a generated answer, decide if the answer is well-supported by the context. "
                    "Reply with exactly one word: 'high' or 'low'."
                ),
            },
            {
                "role": "user",
                "content": (
                    f"Question: {query}\n\nContext:\n{context}\n\nAnswer: {answer}\n\n"
                    "Is this answer well-supported by the context? Reply 'high' or 'low'."
                ),
            },
        ],
        temperature=0,
        max_tokens=5,
    )
    verdict = resp.choices[0].message.content.strip().lower()
    return "high" if "high" in verdict else "low"


def refine_query(original_query: str, previous_answer: str, context: str) -> str:
    """Generate a more targeted query based on what was missing from previous retrieval."""
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a search query optimizer. Given an original question, a previous "
                    "attempt at answering it, and the retrieved context, generate a more specific "
                    "search query that would retrieve better information. Return only the query."
                ),
            },
            {
                "role": "user",
                "content": (
                    f"Original question: {original_query}\n"
                    f"Previous answer (low confidence): {previous_answer}\n"
                    "Retrieved context was insufficient. Write a better search query:"
                ),
            },
        ],
        temperature=0.3,
        max_tokens=100,
    )
    return resp.choices[0].message.content.strip()


def self_rag(query: str, index, max_iterations: int = 2, top_k: int = 5) -> SelfRAGResult:
    """
    Self-RAG loop:
    1. Retrieve context
    2. Generate answer
    3. Check confidence — if low and iterations remain, refine query and retry
    """
    queries_used = [query]
    current_query = query
    answer = ""

    for iteration in range(1, max_iterations + 1):
        results = retrieve(current_query, index, top_k=top_k)
        context = "\n\n---\n\n".join(r.text for r in results)
        answer = generate_answer(query, context)  # Always answer the original question
        confidence = check_answer_confidence(query, context, answer)

        if confidence == "high" or iteration == max_iterations:
            return SelfRAGResult(
                answer=answer,
                iterations=iteration,
                final_confidence=confidence,
                queries_used=queries_used,
            )

        refined = refine_query(query, answer, context)
        queries_used.append(refined)
        current_query = refined

    return SelfRAGResult(answer=answer, iterations=max_iterations, final_confidence="low", queries_used=queries_used)


def retrieval_quality_score(results: list[RetrievalResult]) -> float:
    """Average similarity score. Low score means the query is far from indexed content."""
    if not results:
        return 0.0
    return sum(r.score for r in results) / len(results)


def web_search_fallback(query: str) -> list[RetrievalResult]:
    """
    Placeholder for real web search (Brave, Tavily, Serper, etc.).
    In production: call search API, fetch top pages, chunk and return.
    """
    return [RetrievalResult(
        text=f"[Web search result for: {query}] — integrate a real search API here.",
        score=0.5,
        metadata={"source": "web", "query": query},
    )]


def corrective_rag(query: str, index, quality_threshold: float = 0.4, top_k: int = 5) -> dict:
    """
    Corrective RAG: evaluate retrieval quality before generation.
    Falls back to web search if index retrieval quality is too low.
    """
    results = retrieve(query, index, top_k=top_k)
    quality = retrieval_quality_score(results)
    used_fallback = False

    if quality < quality_threshold:
        results = web_search_fallback(query)
        used_fallback = True

    context = "\n\n---\n\n".join(r.text for r in results)
    answer = generate_answer(query, context)

    return {
        "answer": answer,
        "retrieval_quality": quality,
        "used_fallback": used_fallback,
        "sources": [r.metadata for r in results],
    }

Common mistakes

1. Unlimited self-RAG loops. Without a max iteration cap, a low-confidence system will loop indefinitely. Always set max_iterations=2 or max_iterations=3 as an upper bound. Most quality gains happen in the first retry.

2. Re-querying with the same query. If the first retrieval was poor, re-retrieving with the identical query returns the same results. Always refine the query before re-retrieval — that is the entire mechanism that makes self-RAG useful.

3. Using self-RAG for every query. The extra LLM calls add 200-800ms and cost. Apply self-RAG selectively: only for queries where the stakes are high, or where initial retrieval quality is low. Gate it with a retrieval quality score first.

4. Treating all fallbacks as equivalent. Web search returns unstructured, potentially irrelevant, potentially outdated content. Log clearly when fallback was used so you can evaluate whether fallback answers are actually better.

5. Multi-hop without dependency tracking. If sub-question B depends on the answer to sub-question A, running them in parallel produces wrong inputs. Map the dependency graph before scheduling retrieval.

Try it yourself

Take a RAG system you have built (or the one from the previous lesson). Write 10 test questions: 5 that your system answers well and 5 that it answers poorly (wrong, incomplete, or hallucinated). For the 5 poor answers, identify which pattern would fix each: multi-hop (question requires combining multiple sources), self-RAG (answer is generated but not well-supported), or corrective RAG (retrieval quality is low). Implement self-RAG for one of the poor cases and compare the output with and without the confidence-checking loop.