Your personal operating system for career transition.

Transform JSONL Trace Logs into Aggregated Metrics

AI services emit structured logs for every LLM call: model used, input/output token counts, latency, feature name, and whether the call succeeded. These logs are the raw material for operational dashboards, cost attribution, and debugging regressions.

What to build

Implement aggregate_traces(lines: list[str]) -> AggregateReport that parses JSONL trace logs and produces a structured report.

Input format — each line is a JSON object with:

• request_id: str
• model: str
• feature: str
• input_tokens: int
• output_tokens: int
• latency_ms: int
• success: bool
• error: str | null

AggregateReport dataclass fields:

• total_calls: int
• success_count: int
• failure_count: int
• success_rate: float — rounded to 4 decimal places
• avg_latency_ms: float — across all calls
• p95_latency_ms: float — 95th percentile latency
• total_input_tokens: int
• total_output_tokens: int
• by_model: dict[str, ModelStats] — per-model breakdown
• by_feature: dict[str, FeatureStats] — per-feature breakdown

ModelStats: call_count, success_rate, avg_latency_ms, total_tokens

FeatureStats: call_count, success_rate, avg_latency_ms, avg_tokens_per_call

Why this matters

Operational insight from trace data is one of the most direct ways to improve an AI system. Engineers who can write this transformation quickly can answer production questions in minutes instead of filing tickets to a data team.

Constraints

• Skip malformed lines (log a warning, do not abort).
• p95_latency_ms should use numpy-free percentile computation.
• All float fields rounded to 2 decimal places except success_rate (4 places).