Agent memory and state

Why this matters

LLMs are stateless. Every API call starts with no memory of previous interactions — you supply the entire context in the messages array. For short conversations this is fine. For agents that run multi-step tasks, interact over hours, or serve returning users, memory management becomes a core engineering problem.

Without memory management, your agent will hit context window limits mid-task, lose track of earlier decisions, or spend most of its token budget re-reading old messages. In production, you will encounter all three problems. The techniques in this lesson — sliding windows, summarization, and persistence — are how you keep agents functional and affordable over long interactions.

Core concepts

Context window budgeting. Every model has a context limit (128K tokens for GPT-4o, 200K for Claude). Your budget splits into: system prompt, memory/history, current tool results, and space for the model's response. If history consumes 90% of your window, the model has no room to think. Budget explicitly.

Sliding window. Keep the most recent N messages and drop older ones. Simple, fast, and works well for conversations where recent context matters most. The risk is losing important early context — the user's original goal, key decisions, or constraints stated at the start.

Summarization. Periodically compress older messages into a summary. The agent sees: [summary of turns 1-20] + [full turns 21-30]. This preserves important context while staying within token limits. The tradeoff is that summarization itself costs tokens and can lose details.

Sliding window + summary (hybrid). The most practical production pattern. Keep recent messages in full, maintain a running summary of everything before that, and include key facts extracted from the conversation. This gives the model both detailed recent context and compressed long-term memory.

Persistence. For agents that serve returning users, store conversation summaries and extracted facts in a database. On the next session, load the relevant state into the system prompt. This creates continuity across sessions without replaying entire conversation histories.

Working example

Here is a memory manager that implements the hybrid sliding window + summarization pattern with explicit token budgeting.

import tiktoken
from dataclasses import dataclass, field
from openai import OpenAI

client = OpenAI()
encoder = tiktoken.encoding_for_model("gpt-4o")

def count_tokens(text: str) -> int:
    return len(encoder.encode(text))

def count_message_tokens(messages: list[dict]) -> int:
    return sum(count_tokens(m.get("content", "")) for m in messages)

@dataclass
class MemoryConfig:
    max_context_tokens: int = 120_000   # Leave headroom below model limit
    system_prompt_tokens: int = 500      # Reserved for system prompt
    response_tokens: int = 4_000         # Reserved for model output
    recent_window: int = 10              # Keep last N messages in full
    summary_trigger: int = 15            # Summarize when history exceeds this

@dataclass
class AgentMemory:
    config: MemoryConfig = field(default_factory=MemoryConfig)
    messages: list[dict] = field(default_factory=list)
    summary: str = ""
    key_facts: list[str] = field(default_factory=list)
    total_tokens_used: int = 0

    @property
    def available_tokens(self) -> int:
        reserved = self.config.system_prompt_tokens + self.config.response_tokens
        return self.config.max_context_tokens - reserved

    def add_message(self, role: str, content: str):
        self.messages.append({"role": role, "content": content})
        self.total_tokens_used += count_tokens(content)

        # Trigger summarization when history gets long
        if len(self.messages) > self.config.summary_trigger:
            self._compress()

    def _compress(self):
        """Summarize older messages, keep recent window."""
        if len(self.messages) <= self.config.recent_window:
            return

        old_messages = self.messages[:-self.config.recent_window]
        old_text = "\n".join(
            f"{m['role']}: {m['content']}" for m in old_messages
        )

        # Ask the model to summarize and extract key facts
        response = client.chat.completions.create(
            model="gpt-4o-mini",  # Use a cheap model for summarization
            messages=[
                {
                    "role": "system",
                    "content": (
                        "Summarize the following conversation segment. "
                        "Preserve: decisions made, user preferences, task progress, "
                        "and any constraints. Output JSON with keys: "
                        "summary (string), key_facts (list of strings)."
                    ),
                },
                {"role": "user", "content": old_text},
            ],
            temperature=0.1,
        )

        import json
        try:
            result = json.loads(response.choices[0].message.content)
            self.summary = result.get("summary", "")
            new_facts = result.get("key_facts", [])
            # Deduplicate key facts
            for fact in new_facts:
                if fact not in self.key_facts:
                    self.key_facts.append(fact)
        except json.JSONDecodeError:
            self.summary = response.choices[0].message.content

        # Keep only the recent window
        self.messages = self.messages[-self.config.recent_window:]

    def build_context(self, system_prompt: str) -> list[dict]:
        """Build the full message list for the next API call."""
        context = [{"role": "system", "content": system_prompt}]

        # Inject memory context if we have it
        if self.summary or self.key_facts:
            memory_block = "## Conversation Memory\n"
            if self.summary:
                memory_block += f"**Summary of earlier conversation:**\n{self.summary}\n\n"
            if self.key_facts:
                memory_block += "**Key facts:**\n"
                memory_block += "\n".join(f"- {f}" for f in self.key_facts)
            context.append({"role": "system", "content": memory_block})

        # Add recent messages
        context.extend(self.messages)

        # Final token check — trim if over budget
        while count_message_tokens(context) > self.available_tokens and len(context) > 2:
            context.pop(1)  # Remove oldest non-system message

        return context

    def get_stats(self) -> dict:
        context = self.build_context("test")
        return {
            "messages_in_window": len(self.messages),
            "summary_length": count_tokens(self.summary),
            "key_facts": len(self.key_facts),
            "context_tokens": count_message_tokens(context),
            "available_tokens": self.available_tokens,
            "total_tokens_used": self.total_tokens_used,
        }

# Usage
memory = AgentMemory()
memory.add_message("user", "I want to build a RAG system for legal documents.")
memory.add_message("assistant", "Great choice. What document formats will you ingest?")
memory.add_message("user", "PDFs and Word docs, about 10,000 documents total.")
# ... after many more turns, older messages get summarized automatically

stats = memory.get_stats()
print(f"Context tokens: {stats['context_tokens']} / {memory.available_tokens}")

The key design choices: summarization uses a cheap model (gpt-4o-mini) to save cost, key facts are deduplicated and persist across summaries, and there is a hard token budget enforced at build time.

Common mistakes

1. No token budgeting. You send the full history until the API returns a context length error. By then you have already paid for the failed request.
2. Summarizing with the expensive model. Summarization is a low-stakes task. Use a cheap, fast model and save your budget for the agent's actual reasoning.
3. Losing the user's original goal. If the user's first message states the objective, make sure it survives summarization — either as a key fact or by always including the first message.
4. No persistence layer. When the agent restarts, all memory is lost. Store summaries and key facts in a database keyed by session or user ID.
5. Over-summarizing. If you summarize every 5 messages, you spend as much on summarization as on the actual task. Find the right trigger threshold for your use case.

Try it yourself

Build a chatbot with the hybrid memory manager above. Have a 30-turn conversation with it about a technical topic, then check the stats. Can the agent still recall decisions from turn 3 at turn 30? Experiment with different recent_window and summary_trigger values to find the sweet spot for your use case.