Token budgeting: the engineer's guide to context window management
The context window is the most important finite resource in an LLM application. Ignoring it produces a class of bugs that are hard to diagnose: the model's response quality degrades silently as context fills up, then crashes with a cryptic 400 error when it overflows.
This article gives you the mental model and practical tools to manage token budgets deliberately.
Why context windows matter more than you think
Most engineers think about token limits as a constraint to avoid hitting. The more useful framing: the context window is a shared workspace. Everything you put in it competes for the model's attention. A bloated system prompt, a long conversation history, a full document dump — each one crowds out the others.
Research on large context models consistently shows that retrieval quality degrades on tokens in the middle of a long context (the "lost in the middle" effect). This means that stuffing your full context budget is not just expensive — it can actively hurt answer quality compared to a well-trimmed, focused context.
The budget allocation model
Treat the context window like memory allocation: decide upfront how much each component gets.
Example allocation for a 16K token window:
System prompt: 500-1000 tokens (~6%)
User input: 500-2000 tokens (variable, cap at max)
Retrieved context: 4000-6000 tokens (~35%)
Conversation history: 2000-4000 tokens (~20%)
Output reserve: 2000-4000 tokens (~20%)
Buffer/overhead: ~500 tokens (~3%)
The output reserve is often forgotten. If your prompt consumes 15,500 tokens of a 16K window, your max_tokens=1024 response will be truncated at 500. Always subtract the output budget from the input side.
Counting tokens accurately
OpenAI and Anthropic both provide token counting utilities. For OpenAI-compatible models:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base") # GPT-4o, GPT-4 family
# For Claude: use Anthropic's tokenizer or the ~4 chars/token approximation
def count_prompt_tokens(messages: list[dict]) -> int:
"""Count tokens for an OpenAI-format messages array."""
total = 3 # every reply is primed with <|start|>assistant<|message|>
for msg in messages:
total += 4 # role + message boundary tokens
total += len(enc.encode(msg.get("content") or ""))
return total
For Anthropic, the SDK provides client.count_tokens(messages=messages, system=system) that returns exact token counts.
Priority order for trimming
When your assembled prompt exceeds the budget, trim in this order (least important first):
- Oldest conversation history — the current turn is always most important
- Lowest-scoring retrieved chunks — keep the most relevant context
- Optional context sections — if your system prompt has an optional "user preferences" block, make it conditionally included
- System prompt verbosity — if the system prompt has been growing organically, audit and trim it
Never trim: the current user message, the top-ranked retrieved chunks, or critical instruction blocks that define the model's behavior.
Practical budgeting patterns
Hard budget enforcement before the call:
MAX_CONTEXT_TOKENS = 12000
OUTPUT_RESERVE = 2000
CONTEXT_BUDGET = MAX_CONTEXT_TOKENS - OUTPUT_RESERVE # 10000
def enforce_budget(messages: list[dict]) -> list[dict]:
while count_prompt_tokens(messages) > CONTEXT_BUDGET and len(messages) > 1:
# Find the oldest non-system message and remove it
for i, msg in enumerate(messages):
if msg["role"] != "system":
messages.pop(i)
break
return messages
Soft budget warnings in development:
import logging
logger = logging.getLogger(__name__)
def assemble_and_warn(messages: list[dict], budget: int) -> list[dict]:
tokens = count_prompt_tokens(messages)
if tokens > budget * 0.9:
logger.warning(
"context_budget_warning",
extra={"tokens": tokens, "budget": budget, "pct": tokens / budget},
)
return messages
Soft warnings during development surface budget issues before they become hard failures in production.
Common failures and what they mean
| Symptom | Likely cause |
|---|---|
| 400 "context length exceeded" | No token budget enforcement |
| Responses ignore early conversation | History trimmed too aggressively, or "lost in the middle" effect |
| Response quality degrades over long sessions | Context budget never enforced, model attention diluted |
| High costs on simple features | System prompt or history bloat — not trimming unused context |
| Truncated responses | Output reserve not subtracted from input budget |
The bottom line
Token budgeting is not an optimization — it is a correctness requirement. Every production LLM feature should have an explicit budget allocation, a counting function that measures it, and a trim strategy that executes before the API call. Features that skip this step work fine in demos and break mysteriously in production.