Your personal operating system for career transition.

Count tokens and trim a messages array to a budget

Context window management is the most common source of silent failures in production LLM features. When history grows too long, the API returns a 400 error at the worst possible moment.

What you are building

Implement two functions:

count_messages_tokens(messages: list[dict]) -> int

Count the total tokens in a messages array. Use tiktoken with the cl100k_base encoding. Add 4 tokens per message for role overhead.

trim_to_budget(messages: list[dict], max_tokens: int, protect_last: int = 1) -> list[dict]

Given a messages list that may exceed max_tokens, return a trimmed list that fits within the budget. Rules:

• Always protect the last protect_last messages (the most recent turns)
• Drop oldest messages first until the list fits
• Never drop below protect_last messages (return at minimum the protected ones)
• Return the list in original order (not reversed)

Requirements

• Use tiktoken.get_encoding("cl100k_base") for token counting
• Each message costs len(tokens) + 4 tokens (4 for role/separator overhead)
• trim_to_budget must never modify the input list (return a new list)

Example

messages = [
    {"role": "system", "content": "You are helpful."},  # ~20 tokens
    {"role": "user", "content": "First question?"},      # ~15 tokens
    {"role": "assistant", "content": "First answer."},   # ~15 tokens
    {"role": "user", "content": "Second question?"},     # ~15 tokens
]
trimmed = trim_to_budget(messages, max_tokens=50, protect_last=1)
# Should keep the system message and most recent user message