Your personal operating system for career transition.

Build an Observability Metrics Aggregator

Production LLM services emit request events. An aggregator processes them into the operational metrics that matter: latency percentiles, error rates, cost by feature, quality trend.

What to build

LLMObservabilityAggregator processing RequestEvent(timestamp, feature, model, latency_ms, input_tokens, output_tokens, success, quality_score):

1. ingest(event)
2. latency_percentiles(feature=None) -> dict — {"p50", "p95", "p99"} in ms
3. error_rate(window_minutes=60) -> float
4. cost_by_feature(model_prices: dict) -> dict
5. quality_trend(bucket_minutes=60) -> list[dict] — [{"bucket_start", "avg_score", "count"}] oldest-first, only scored events

Constraints

• Standard library only.
• Nearest-rank percentile: ceil(p/100 * n) - 1 index into sorted array.