AI Engineer Portal

Prompt, context, tools, and memory

Why this matters

Every LLM feature you build is a message array plus configuration. Understanding what goes into that array — and why — is the foundational skill that separates engineers who debug prompts by intuition from engineers who diagnose them systematically. Frameworks like LangChain, LlamaIndex, and the Anthropic SDK are all doing the same thing underneath: assembling messages, managing context, and routing tool calls. If you understand the primitives, you can reason about any framework.

Core concepts

The message array. LLM APIs are not magic chat interfaces — they accept a list of messages, each with a role and content. The three roles that matter in practice are:

system: Instructions, persona, and constraints that shape every response. The model treats this as ground truth about how to behave.
user: The human turn — typically the user's request or input.
assistant: Prior model responses. When you include these in the messages array, you are giving the model memory of what it already said.

A complete request looks like:

messages = [
    {"role": "system", "content": "You are a helpful code reviewer..."},
    {"role": "user", "content": "Can you review this function?"},
    {"role": "assistant", "content": "Sure, let me look at it..."},
    {"role": "user", "content": "Here is the code: ..."},
]

The model sees all of these messages simultaneously, not sequentially. Context is not a stream of consciousness — it is a snapshot.

Context windows and budget management. Every model has a context window: the maximum number of tokens it can process in a single call. As of 2025, major models support 128K to 1M tokens, but large contexts are slow and expensive. In production, you budget each component:

System prompt: 200–1000 tokens for most features
Conversation history: variable, needs trimming
Retrieved context (RAG): 2000–8000 tokens
User input: variable
Output budget: reserve space for the response

Tool/function calling schemas. Tools let the model invoke external functions — search, database queries, API calls. The model does not run code; it outputs a structured request, and your application executes the call and returns the result.

The OpenAI format:

tools = [{
    "type": "function",
    "function": {
        "name": "search_orders",
        "description": "Search customer orders by order ID or customer email. Use when the user asks about order status.",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Order ID or customer email"},
                "status_filter": {
                    "type": "string",
                    "enum": ["pending", "shipped", "delivered"],
                    "description": "Optional status filter"
                }
            },
            "required": ["query"]
        }
    }
}]

The Anthropic format uses the same JSON Schema but wraps it slightly differently:

tools = [{
    "name": "search_orders",
    "description": "Search customer orders by order ID or customer email.",
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string"}
        },
        "required": ["query"]
    }
}]

Conversation memory patterns. LLMs are stateless — they have no memory between API calls unless you pass it to them. Four practical patterns:

Full history: Pass every previous message. Simple but expensive; hits limits quickly.
Sliding window: Keep the last N turns. Cheap but loses early context.
Summarize on overflow: When history exceeds a token budget, call the model to compress old turns into a summary, then insert the summary as a system message.
Persistent structured memory: Store facts (user preferences, past decisions) in a database and inject them into the system prompt at call time.

Working example

Here is a prompt assembly function that combines all four components into a properly budgeted messages array:

from typing import TypedDict
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

class Message(TypedDict):
    role: str
    content: str

def assemble_prompt(
    system_prompt: str,
    user_input: str,
    history: list[Message],
    retrieved_context: list[str],
    tools: list[dict],
    max_context_tokens: int = 12000,
    output_reserve_tokens: int = 2000,
) -> list[Message]:
    """
    Assemble a messages array with token budget management.
    Priority: system > user input > retrieved context > history (oldest dropped first)
    """
    available = max_context_tokens - output_reserve_tokens

    # Always include system prompt
    system_tokens = count_tokens(system_prompt)
    available -= system_tokens

    # Always include current user input
    user_tokens = count_tokens(user_input)
    available -= user_tokens

    # Add retrieved context if provided
    context_text = ""
    if retrieved_context:
        context_text = "\n\n".join(f"<context>\n{c}\n</context>" for c in retrieved_context)
        ctx_tokens = count_tokens(context_text)
        if ctx_tokens <= available:
            available -= ctx_tokens
        else:
            # Truncate context to fit
            context_text = context_text[:available * 4]  # rough chars estimate
            available = 0

    # Fit as much history as possible (newest first, then reverse)
    trimmed_history: list[Message] = []
    for msg in reversed(history):
        msg_tokens = count_tokens(msg["content"]) + 4  # 4 for role overhead
        if msg_tokens <= available:
            trimmed_history.insert(0, msg)
            available -= msg_tokens
        else:
            break  # oldest messages dropped first

    # Assemble final system content
    system_content = system_prompt
    if context_text:
        system_content = system_prompt + "\n\n## Relevant context\n" + context_text

    messages: list[Message] = [{"role": "system", "content": system_content}]
    messages.extend(trimmed_history)
    messages.append({"role": "user", "content": user_input})

    return messages


# Example usage
messages = assemble_prompt(
    system_prompt="You are a helpful customer support assistant for an e-commerce platform.",
    user_input="What is the status of my order ORD-12345?",
    history=[
        {"role": "user", "content": "Hi, I need help with my recent purchase."},
        {"role": "assistant", "content": "Of course! What is your order number?"},
    ],
    retrieved_context=["Order ORD-12345 was shipped on March 15 via FedEx. Tracking: 9400111899223387644924"],
    tools=[],  # pass tool schemas here
)

The key insight: budget management happens at assembly time, not at the provider call. By the time you call the API, you know exactly how many tokens you are spending.

Common mistakes

No token counting at assembly time. Most teams discover context overflow in production when a power user with a long conversation history hits a 400 error. Count tokens before every call.
System prompt bloat. System prompts grow organically over months. A 4000-token system prompt eating 30% of your context budget on every call is expensive. Audit and trim quarterly.
Putting untrusted user input in the system role. User-controlled text in the system role is a prompt injection surface. Always keep untrusted input in the user role. The system role is for your instructions only.
Dropping the last user message. When trimming history for budget, engineers sometimes forget to protect the current user input. The most recent message is the most important one.
Tool schema descriptions that are too vague. Descriptions like "searches the database" do not tell the model when to call the tool versus another tool. Descriptions are load-bearing instructions.

Try it yourself

Build a version of assemble_prompt that supports a fourth content component: structured user profile data (name, past topics, preferences). This should be injected into the system prompt in a clearly labeled block. Test it with a conversation history that is too long to fit and confirm that history trimming preserves the most recent messages.