AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Build a human feedback collection pipeline

Build a Human Feedback Collection Pipeline

User feedback is the ground truth that calibrates all other evaluation signals. Build a feedback collection and aggregation pipeline that captures thumbs ratings and explicit annotations, then surfaces quality insights.

What you are building

Implement a FeedbackPipeline that:

Records thumbs feedback -- stores (trace_id, feature_name, rating: "up"/"down"/"neutral", timestamp_ms).
Records explicit annotations -- stores (trace_id, feature_name, rater_id, helpfulness: 1-5, faithfulness: 1-5, notes: str).
Computes satisfaction rate -- thumbs_up / (thumbs_up + thumbs_down) per feature, ignoring neutral.
Computes annotation quality -- average helpfulness and faithfulness per feature, pass rate (score >= 3).
Identifies low-quality traces -- returns trace IDs where average annotation score < a configurable threshold.
Generates a combined report -- merges thumbs and annotation signals into one dict per feature.

Constraints

Use only the Python standard library.
Features with zero thumbs or zero annotations should report None for those metrics rather than 0.
The low_quality_traces method should accept a min_raters parameter (default 1).

Evaluation / medium / Step 13 of 36

Practice stage

Evaluation and review loops

Hint

Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.

Next drill

Implement A/B testing for model versions

Success criteria

- Produces a useful signal, not decorative output
- Makes regression review easier
- Would support a benchmark or observability loop

Review checklist

- Would this output help decide what to fix next?
- Are important failure modes visible?
- Does the score hide any ambiguity I should record?

from __future__ import annotations
from dataclasses import dataclass

@dataclass
class ThumbsFeedback:
    trace_id: str
    feature_name: str
    rating: str  # "up", "down", "neutral"
    timestamp_ms: int

@dataclass
class Annotation:
    trace_id: str
    feature_name: str
    rater_id: str
    helpfulness: int   # 1-5
    faithfulness: int  # 1-5
    notes: str = ""

class FeedbackPipeline:
    def __init__(self):
        raise NotImplementedError

def record_thumbs(self, feedback: ThumbsFeedback) -> None:
        raise NotImplementedError

def record_annotation(self, annotation: Annotation) -> None:
        raise NotImplementedError

def satisfaction_rate(self, feature_name: str) -> float | None:
        raise NotImplementedError

def annotation_quality(self, feature_name: str) -> dict | None:
        raise NotImplementedError

def low_quality_traces(self, feature_name: str, threshold: float = 2.5, min_raters: int = 1) -> list[str]:
        raise NotImplementedError

def combined_report(self, feature_name: str) -> dict:
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.