AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Trace an agent decision chain

Trace an Agent Decision Chain

A bad final answer might trace back to a wrong tool call in step 2, caused by a misinterpreted instruction in step 1. You need a trace structure capturing every decision point to reconstruct exactly what happened.

What to build

Implement AgentTracer:

start_trace(trace_id, input) — Raises ValueError if trace_id already exists.
record_step(trace_id, step_type, content: dict) — step_type: "thought", "tool_call", "tool_result", "generation", "final_answer". Auto-assigns step_number (1-indexed) and timestamp.
finish_trace(trace_id, success, final_answer=None) -> dict — Returns full trace with "trace_id", "input", "steps", "success", "final_answer", "total_steps", "tool_calls_made", "duration_s".
get_trace(trace_id) -> dict | None
summarize_traces() -> dict — {"total_traces", "successful", "avg_steps", "avg_tool_calls"} across finished traces.

Constraints

Standard library only. time.time() for timestamps.

Evaluation / hard / Step 25 of 36

Practice stage

Evaluation and review loops

Hint

Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.

Next drill

Build a cost and latency dashboard aggregator

Success criteria

- Produces a useful signal, not decorative output
- Makes regression review easier
- Would support a benchmark or observability loop

Review checklist

- Would this output help decide what to fix next?
- Are important failure modes visible?
- Does the score hide any ambiguity I should record?

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.