AI Engineer Portal
Your personal operating system for career transition.
Exercise
Implement structured logging for AI call traces
Implement Structured Logging for AI Call Traces
Debugging bad AI outputs requires the full call context as a queryable structured event: prompt version, model, latency, token usage, validation result.
What to build
Implement AICallLogger:
-
log_call(request_id, model, prompt_version, latency_ms, input_tokens, output_tokens, finish_reason, success, validation_error=None, feature=None)— INFO for success, WARNING for validation errors ("ai_call_validation_error"), ERROR for failures ("ai_call_failed"). -
log_retrieval(request_id, query, num_chunks_retrieved, top_chunk_score, retrieval_ms, filter_criteria=None)— Emits"retrieval_ok". -
log_judge_result(request_id, metric, score, rationale, judge_model, judge_latency_ms)— Emits"eval_judge_result". -
recent_events(n=10) -> list[dict]— Last n events from in-memory buffer.
Constraints
- Use
logger.log(level, event_name, extra=fields)pattern. - Buffer with
deque(maxlen=1000).
Evaluation / medium / Step 24 of 36
Evaluation and review loops
Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.
- - Produces a useful signal, not decorative output
- - Makes regression review easier
- - Would support a benchmark or observability loop
- - Would this output help decide what to fix next?
- - Are important failure modes visible?
- - Does the score hide any ambiguity I should record?
Practice
Generate a variation
Generate a new exercise variation to deepen understanding or practice a related concept.
Attempt history
Recent submissions
Before you submit, decide what a strong answer should make obvious to the reviewer.
No attempts yet.