AI Engineer Portal
Your personal operating system for career transition.
Exercise
Build a cost and latency dashboard aggregator
Build a Cost and Latency Dashboard Aggregator
Without visibility into cost per feature and per model, you cannot make rational optimization decisions for production LLM services.
What to build
Implement CostLatencyDashboard:
-
record(feature, model, input_tokens, output_tokens, latency_ms, success)— Ingests one request event. -
cost_by_feature(model_prices: dict) -> dict—model_prices:{model: {"input_per_1k": float, "output_per_1k": float}}. Returns{feature: {"total_cost_usd", "request_count", "avg_cost_per_request"}}. -
latency_summary(feature: str | None = None) -> dict— Returns{"p50_ms", "p95_ms", "p99_ms", "avg_ms", "error_rate"}. None = global. -
top_cost_drivers(model_prices, n=5) -> list[dict]— Top n(feature, model)pairs by cost. Each:{"feature", "model", "total_cost_usd", "request_count"}, sorted descending.
Constraints
- Standard library only. Nearest-rank percentile:
index = ceil(p/100 * len(data)) - 1.
Evaluation / medium / Step 26 of 36
Evaluation and review loops
Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.
- - Produces a useful signal, not decorative output
- - Makes regression review easier
- - Would support a benchmark or observability loop
- - Would this output help decide what to fix next?
- - Are important failure modes visible?
- - Does the score hide any ambiguity I should record?
Practice
Generate a variation
Generate a new exercise variation to deepen understanding or practice a related concept.
Attempt history
Recent submissions
Before you submit, decide what a strong answer should make obvious to the reviewer.
No attempts yet.