AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Enforce per-request token budgets

Enforce Per-Request Token Budgets

LLM API costs are linear with token usage. Without enforcement, a single user pasting a 50,000-token document into a chat box can generate a $1-$5 API call. Multiply by thousands of users and you have a runaway cost event.

Token budget enforcement happens at prompt construction time — before the API call, not after. This exercise builds the enforcement layer.

What to build

Implement a PromptBudgetEnforcer that:

Takes a max_total_tokens: int and model: str to construct a budget-aware prompt.
build(system: str, history: list[dict], context_chunks: list[str], user_query: str) -> BuiltPrompt — assembles the prompt while respecting the budget.
- System prompt is always included first (it is always high-priority).
- User query is always included (it is what the user asked).
- History is included newest-first until the budget is hit.
- Context chunks are included in order until the budget is hit.
- If any component is truncated, BuiltPrompt.truncated: bool is True and BuiltPrompt.truncated_components: list[str] lists what was cut.
Use a simple token approximation: 1 token = 4 characters.
BuiltPrompt should contain the final messages: list[dict] ready to send to the API, plus budget metadata.
Add a budget_remaining(built: BuiltPrompt) -> int method showing how many tokens are left for the model's response.

Constraints

No external tokenizer libraries — use the 4-char approximation.
max_total_tokens includes both the prompt tokens and the expected completion. Reserve 20% for completion.
Raise ValueError if the system prompt + user query alone exceed the budget.

Api Async / medium / Step 10 of 23

Practice stage

Async and provider control

Hint

Make waiting behavior explicit. Timeouts, retries, and concurrency limits matter more than squeezing everything into one helper.

Next drill

Build a rate limiter for AI API endpoints

Success criteria

- Uses async boundaries coherently
- Makes timeout and retry decisions legible
- Would be maintainable under provider instability

Review checklist

- Is timeout behavior explicit?
- Is retryable failure separate from terminal failure?
- Would logs reveal what actually timed out?

from __future__ import annotations
from dataclasses import dataclass, field

@dataclass
class BuiltPrompt:
    messages: list[dict]
    prompt_tokens_approx: int
    max_total_tokens: int
    truncated: bool = False
    truncated_components: list[str] = field(default_factory=list)

class PromptBudgetEnforcer:
    CHARS_PER_TOKEN = 4
    COMPLETION_RESERVE_PCT = 0.20

def __init__(self, max_total_tokens: int, model: str = "gpt-4o-mini"):
        self.max_total_tokens = max_total_tokens
        self.model = model

def _count_tokens(self, text: str) -> int:
        # TODO: approximate token count
        raise NotImplementedError

def _prompt_budget(self) -> int:
        # TODO: compute available tokens for the prompt (reserve 20% for completion)
        raise NotImplementedError

def build(
        self,
        system: str,
        history: list[dict],
        context_chunks: list[str],
        user_query: str,
    ) -> BuiltPrompt:
        # TODO: assemble messages within budget
        raise NotImplementedError

def budget_remaining(self, built: BuiltPrompt) -> int:
        # TODO: return tokens available for model response
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.