AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Implement RAG answer generation with inline citations

Implement RAG Answer Generation with Inline Citations

Most RAG implementations generate an answer and separately display source documents, with no connection between a specific claim in the answer and the specific chunk that supports it. This makes it impossible for users to verify individual facts and impossible for engineers to audit faithfulness at the claim level.

Citation-linked generation solves this: each factual claim in the answer is tagged with the chunk ID it came from using [CITE:chunk_id] notation, enabling both "show source" UI and automated faithfulness checking.

What you are building

Create a CitationRAG class that:

Assembles a prompt that instructs the LLM to cite sources inline using [CITE:chunk_id] after each factual claim.
Parses citations from the generated answer — extract (claim_text, cited_chunk_id) pairs using regex.
Validates citations — verify each cited chunk ID exists in the provided context. Flag phantom citations (citing a non-existent ID).
Returns a CitedAnswer dataclass with: raw answer text, citation map {chunk_id: [claims]}, phantom citations list, and grounding_score (fraction of claims with a valid citation).

Use an injectable mock LLM function for testability.

Why this matters

Citation tracking is the foundation of trustworthy RAG. Without it, you are asking users to trust an answer on faith. With it, you power "show source" UI, audit faithfulness automatically, and identify which retrieval improvements have the highest impact.

Constraints

Use regex to extract [CITE:...] patterns. Type-hint everything.
LLM function signature: Callable[[str], str].

Retrieval / medium / Step 10 of 15

Practice stage

Retrieval quality and ranking

Hint

Do not trust a single score blindly. Combine ranking logic with metadata and think about why a candidate deserves to rise.

Next drill

Build a RAG evaluation harness with Recall@K and MRR

Success criteria

- Ranking rule is explainable
- Supports future iteration without rewriting everything
- Feels tied to product trust rather than pure math trivia

Review checklist

- Did I account for both relevance and metadata?
- Would I be able to explain the ranking rule in an interview?
- Does the code make future tuning easy?

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import Callable

@dataclass
class ContextChunk:
    id: str
    text: str
    metadata: dict = field(default_factory=dict)

@dataclass
class CitedAnswer:
    raw_answer: str
    citation_map: dict[str, list[str]]  # chunk_id -> list of claims citing it
    phantom_citations: list[str]         # chunk IDs cited but not in context
    grounding_score: float               # fraction of claims with valid citation

CITATION_PATTERN = re.compile(r'\[CITE:([\w\-]+)\]')

def build_rag_prompt(question: str, chunks: list[ContextChunk]) -> str:
    """Build prompt with instructions to cite every factual claim with [CITE:chunk_id]."""
    # TODO: format chunks as "[chunk_id]: text", add citation instruction
    raise NotImplementedError

def parse_citations(answer: str) -> list[tuple[str, str]]:
    """
    Extract (claim_text, chunk_id) pairs from answer.
    Empty chunk_id means uncited claim.
    Split on sentence boundaries; each [CITE:id] is paired with preceding claim text.
    """
    # TODO: implement
    raise NotImplementedError

class CitationRAG:
    def __init__(self, llm_fn: Callable[[str], str]):
        self.llm_fn = llm_fn

def answer(self, question: str, chunks: list[ContextChunk]) -> CitedAnswer:
        # TODO: build prompt, generate, parse, validate, compute score
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.