RAG Context Window Management
The context window is a finite resource. Every token you give to retrieved chunks is a token you cannot give to the system prompt, conversation history, or expected output. Getting context assembly wrong silently degrades answer quality without raising any errors.
The three layers of the context budget
A typical RAG prompt has three layers competing for context space:
- System prompt — instructions, persona, safety rules. Usually 100–500 tokens. Fixed cost per request.
- Retrieved context — chunks from retrieval. Variable, needs active management.
- Output space — tokens the model needs to generate the answer. Set via
max_tokens. If you underestimate, the model truncates.
A common mistake is allocating the full context window to retrieved context and then discovering the system prompt and output together exceed what remains.
Token budgeting in practice
effective_context_budget = model_context_limit - system_prompt_tokens - max_output_tokens - safety_margin
For gpt-4o with a 128K context window, a 500-token system prompt, and 1000 output tokens: effective context budget is ~126,500 tokens. That is generous. For gpt-4o-mini with an 128K context: same math, same result. For fine-tuned models or edge deployments with 8K or 16K contexts, the budget is tight.
Deduplication before assembly
Embedding retrieval often returns multiple chunks from the same section of the same document — slightly different phrasings of the same content. Filling your context budget with near-duplicate passages wastes tokens without improving answer quality.
Before assembling context, check each candidate chunk against already-selected chunks using Jaccard similarity or cosine similarity on their embeddings. Drop near-duplicates (>80% word overlap) before counting their tokens against the budget.
The lost-in-the-middle effect
Research consistently shows that LLMs are best at using context that appears at the beginning and end of the context window, and systematically miss information in the middle. For RAG:
- Place the most relevant chunks first and last.
- If you are retrieving 5 chunks, do not place the highest-scored chunk in position 3.
- For synthesis queries across multiple sources, consider interleaving sources rather than placing all chunks from source A before source B.
Citation headers enable grounding
Every chunk in the context should carry a citation header: [Source: document_name, Section: section_name]. This gives the LLM three things: the information it needs to cite sources in its answer, the context to understand where each piece of information came from, and a structural signal that helps separate facts from different sources.
Without citation headers, the model produces answers that are faithful but uncitable, and users cannot verify individual claims.
Practical takeaway
Treat context assembly as a resource allocation problem, not a concatenation problem. Budget explicitly for system prompt and output. Deduplicate before filling the budget. Order by relevance (highest first). Include citation headers. Log which chunks were truncated so you can detect when your token budget is too small for your retrieval quality targets.