Your personal operating system for career transition.

Build a Human-in-the-Loop Annotation Queue

Automated evaluation is fast but imperfect. An annotation queue routes only the cases where human judgment adds the most value: borderline scores, judge disagreements, and new failure modes.

What to build

Implement AnnotationQueue:

1. enqueue(case_id, prediction, reference, auto_score, priority="normal") — Priority: "high", "normal", or "low". Raises ValueError if case_id already exists.

2. next_for_review(reviewer_id) -> dict | None — Returns highest-priority unreviewed case (high > normal > low, then FIFO), marks it "in_review". Returns None if empty.

3. submit_review(case_id, reviewer_id, human_score, notes="") -> dict — Records score, marks "reviewed". Returns {"case_id", "auto_score", "human_score", "delta", "agrees"} where agrees = abs(delta) <= 0.2.

4. stats() -> dict — Returns {"total", "pending", "in_review", "reviewed", "agreement_rate"}.

Constraints

• Standard library only. Use time.time() for enqueue timestamps.