AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Track and alert on LLM cost per feature

Track and Alert on LLM Cost Per Feature

Without cost visibility, you will discover that your most popular feature is generating $30,000/month in LLM costs via the monthly billing invoice — after the fact. The solution is instrumented cost tracking that captures cost at request time, grouped by feature.

What to build

Implement a CostLedger that:

record(event: CostEvent) -> None — record a cost event with feature, model, input_tokens, output_tokens, user_id.
daily_report(date_str: str | None = None) -> DailyReport — costs grouped by feature for today. Include total cost, per-feature breakdown, and top 3 costliest requests.
check_alerts(budgets: dict[str, float]) -> list[BudgetAlert] — check if any feature has exceeded its daily budget.
hourly_burn_rate(feature: str) -> float — average cost per hour over the last 24 hours for a feature.

Use these rates per 1000 tokens (input/output):

gpt-4o-mini: $0.00015 / $0.0006
gpt-4o: $0.0025 / $0.01
claude-3-5-haiku-20241022: $0.0008 / $0.004

Constraints

Standard library only.
Use datetime for timestamps.
Events stored in-memory.

Evaluation / medium / Step 16 of 36

Practice stage

Evaluation and review loops

Hint

Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.

Next drill

Implement A/B testing for model versions in production

Success criteria

- Produces a useful signal, not decorative output
- Makes regression review easier
- Would support a benchmark or observability loop

Review checklist

- Would this output help decide what to fix next?
- Are important failure modes visible?
- Does the score hide any ambiguity I should record?

from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime, timezone
from collections import defaultdict

MODEL_COSTS = {
    "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
    "gpt-4o": {"input": 0.0025, "output": 0.01},
    "claude-3-5-haiku-20241022": {"input": 0.0008, "output": 0.004},
}

@dataclass
class CostEvent:
    feature: str
    model: str
    input_tokens: int
    output_tokens: int
    user_id: str | None = None
    timestamp: datetime = field(default_factory=lambda: datetime.now(timezone.utc))

@property
    def cost_usd(self) -> float:
        # TODO: compute cost using MODEL_COSTS
        raise NotImplementedError

@dataclass
class DailyReport:
    date: str
    total_usd: float
    by_feature: dict[str, float]
    top_requests: list[dict]

@dataclass
class BudgetAlert:
    feature: str
    spent_usd: float
    budget_usd: float
    overage_pct: float

class CostLedger:
    def __init__(self):
        self._events: list[CostEvent] = []

def record(self, event: CostEvent) -> None:
        raise NotImplementedError

def daily_report(self, date_str: str | None = None) -> DailyReport:
        raise NotImplementedError

def check_alerts(self, budgets: dict[str, float]) -> list[BudgetAlert]:
        raise NotImplementedError

def hourly_burn_rate(self, feature: str) -> float:
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.