Why this matters

"The model is ignoring the instructions." These are context engineering failures — frustrating to debug because the problem is in the assembled context you are sending to the model, not in the code itself.

Context debugging is a systematic practice with known failure modes, measurable quality signals, and concrete diagnostic steps.

Core concepts

How context failures manifest

• Instruction dilution: the model follows instructions inconsistently as conversation length grows
• Context pollution: irrelevant content misdirects the model — a retrieved document that was semantically adjacent but factually wrong for the query
• Recency bias: the model disproportionately attends to content near the end of the context
• Lost-in-the-middle: content in the middle of a long context receives less attention than content at the beginning and end

Context relevance scoring

Score each retrieved chunk against the query before including it.

import math
from typing import Callable


def cosine_similarity(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x ** 2 for x in a))
    norm_b = math.sqrt(sum(x ** 2 for x in b))
    if norm_a == 0 or norm_b == 0:
        return 0.0
    return dot / (norm_a * norm_b)


async def score_context_relevance(
    query: str,
    context_chunks: list[str],
    embed_fn: Callable[[str], list[float]],
    threshold: float = 0.6,
) -> list[tuple[str, float]]:
    query_embedding = await embed_fn(query)
    results = []
    for chunk in context_chunks:
        chunk_embedding = await embed_fn(chunk)
        results.append((chunk, cosine_similarity(query_embedding, chunk_embedding)))
    return sorted(
        [(c, s) for c, s in results if s >= threshold], key=lambda x: x[1], reverse=True
    )

Instruction dilution diagnosis

Run your system prompt plus a test query at different history lengths (0, 5, 10, 20 turns). Measure whether a specific constraint is followed at each length. If compliance drops, you have instruction dilution.

Mitigations: re-inject critical instructions before the final user message; summarize history more aggressively; use shorter system prompts.

Code: ContextDebugger

from dataclasses import dataclass, field
from typing import Callable


@dataclass
class ContextReport:
    total_tokens: int
    layer_breakdown: dict[str, int]
    dropped_layers: list[str]
    relevance_scores: dict[str, float]
    warnings: list[str]


@dataclass
class ContextDebugger:
    count_fn: Callable[[str], int] = estimate_tokens
    relevance_threshold: float = 0.6

    def analyze_messages(self, messages: list[dict]) -> ContextReport:
        total, breakdown, warnings = 0, {}, []

        for i, msg in enumerate(messages):
            content = msg.get("content", "")
            tokens = self.count_fn(content)
            breakdown[f"{msg['role']}[{i}]"] = tokens
            total += tokens

        if len(messages) > 6:
            middle_start = len(messages) // 4
            middle_end = 3 * len(messages) // 4
            middle_tokens = sum(
                self.count_fn(messages[i].get("content", ""))
                for i in range(middle_start, middle_end)
            )
            if middle_tokens > total * 0.5:
                warnings.append(
                    f"Lost-in-the-middle risk: {middle_tokens}/{total} tokens "
                    f"({100 * middle_tokens // total}%) are in the middle of context."
                )

        sys_msgs = [m for m in messages if m.get("role") == "system"]
        if sys_msgs and self.count_fn(sys_msgs[0]["content"]) > 2000:
            warnings.append(
                f"System prompt is {self.count_fn(sys_msgs[0]['content'])} tokens "
                "— consider trimming to reduce instruction dilution risk."
            )

        return ContextReport(
            total_tokens=total, layer_breakdown=breakdown,
            dropped_layers=[], relevance_scores={}, warnings=warnings,
        )

    def check_instruction_coverage(
        self, messages: list[dict], required_instructions: list[str]
    ) -> list[str]:
        full_context = " ".join(m.get("content", "") for m in messages)
        return [i for i in required_instructions if i.lower() not in full_context.lower()]

    async def full_debug_report(
        self,
        messages: list[dict],
        query: str,
        retrieved_chunks: list[str],
        embed_fn,
        required_instructions: list[str] | None = None,
    ) -> ContextReport:
        report = self.analyze_messages(messages)

        if retrieved_chunks:
            scored = await score_context_relevance(
                query, retrieved_chunks, embed_fn, self.relevance_threshold
            )
            report.relevance_scores = {f"chunk_{i}": s for i, (_, s) in enumerate(scored)}
            low = [i for i, (_, s) in enumerate(scored) if s < self.relevance_threshold]
            if low:
                report.warnings.append(
                    f"Context pollution: {len(low)} chunks below threshold {self.relevance_threshold}."
                )

        if required_instructions:
            missing = self.check_instruction_coverage(messages, required_instructions)
            if missing:
                report.warnings.append(f"Missing instructions in assembled context: {missing}")

        return report

Common mistakes

Only checking the final response. Add context quality checks before the LLM call, not just after.

Not logging context. Log the full assembled context for anomalous responses.

Treating all warnings as errors. Quality signals are indicators, not hard failures.

Not testing at scale. Run your debugger over 50 real conversations, not just one.

Try it yourself

1. Run ContextDebugger.analyze_messages() on a real conversation from your LLM application.
2. Implement a recency_bias_score method that measures the ratio of tokens in the last quarter vs. the first quarter.
3. Write a test that verifies a required instruction is present in every assembled context.