AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Build an LLM call tracer with span export

Build an LLM Call Tracer with Span Export

Observability for LLM applications goes beyond logging. When a user sees a bad response, you need to reconstruct the entire request lifecycle: what was retrieved, what prompt was built, which model was called, what the tokens and latency were, and where time was spent.

Distributed tracing — using traces and spans — is the right abstraction. A trace represents one user request end-to-end. Spans within the trace represent individual steps (retrieval, prompt construction, LLM call, post-processing).

What to build

Implement a RequestTracer that:

Creates a root trace with a trace_id, request_id, and start timestamp.
span(name: str, **attrs) -> SpanContext — returns a context manager that creates a child span capturing start_ms, end_ms, latency_ms, and all keyword attributes.
Spans auto-complete on context manager exit, even if the body raises (record the error).
export() -> TraceExport — returns the complete trace as a structured object.
TraceExport has: trace_id, request_id, total_latency_ms, spans: list[SpanData], llm_calls: list[LLMCallSummary].
SpanContext.add_llm_call(model, input_tokens, output_tokens, latency_ms) — attach an LLM call summary to the active span.

Constraints

Standard library only.
Spans form a flat list (no tree — single level for simplicity).
export() should be callable multiple times without side effects.

Evaluation / medium / Step 15 of 36

Practice stage

Evaluation and review loops

Hint

Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.

Next drill

Track and alert on LLM cost per feature

Success criteria

- Produces a useful signal, not decorative output
- Makes regression review easier
- Would support a benchmark or observability loop

Review checklist

- Would this output help decide what to fix next?
- Are important failure modes visible?
- Does the score hide any ambiguity I should record?

from __future__ import annotations
import time
import uuid
from dataclasses import dataclass, field

@dataclass
class LLMCallSummary:
    model: str
    input_tokens: int
    output_tokens: int
    latency_ms: int

@dataclass
class SpanData:
    span_id: str
    name: str
    start_ms: int
    end_ms: int = 0
    latency_ms: int = 0
    attributes: dict = field(default_factory=dict)
    error: str = ""
    llm_calls: list[LLMCallSummary] = field(default_factory=list)

@dataclass
class TraceExport:
    trace_id: str
    request_id: str
    total_latency_ms: int
    spans: list[SpanData]
    llm_calls: list[LLMCallSummary]

class SpanContext:
    def __init__(self, span: SpanData):
        # TODO: store span
        raise NotImplementedError

def set(self, key: str, value) -> None:
        # TODO: add attribute to span
        raise NotImplementedError

def add_llm_call(self, model: str, input_tokens: int, output_tokens: int, latency_ms: int) -> None:
        # TODO: attach LLM call summary
        raise NotImplementedError

def __enter__(self) -> "SpanContext":
        raise NotImplementedError

def __exit__(self, exc_type, exc_val, exc_tb) -> bool:
        # TODO: record end time and error if any; do not suppress exceptions
        raise NotImplementedError

class RequestTracer:
    def __init__(self, request_id: str | None = None):
        # TODO: initialize trace
        raise NotImplementedError

def span(self, name: str, **attrs) -> SpanContext:
        # TODO: create and register a new span
        raise NotImplementedError

def export(self) -> TraceExport:
        # TODO: build and return TraceExport
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.