AI Engineer Portal

Human-in-the-loop evaluation

Why this matters

Automated metrics can tell you whether a response is grounded in context or matches a reference string. They cannot tell you whether a response was actually helpful to the person who asked. Human judgment remains the ground truth for AI quality, and the best production teams build systematic workflows to capture that judgment and feed it back into the system.

Human-in-the-loop (HITL) evaluation is not a fallback for when automated metrics fail -- it is the calibration source that makes automated metrics trustworthy. Without it, you are optimizing a proxy that may be drifting away from what users actually want.

Core concepts

Annotation workflows. An annotation workflow is the structured process by which human reviewers evaluate AI outputs. Key components:

Task definition: what exactly is the reviewer being asked to judge?
Rubric: the scoring guide that defines what each score value means
Interface: the tool used to present cases and collect ratings
Sampling strategy: which outputs get reviewed?
Reviewer pool: internal team, domain experts, or crowdsourced annotators

A typical annotation task for a Q&A feature:

Question: [shown to reviewer]
Answer: [shown to reviewer]
Context used: [shown to reviewer]

Rate this answer on:
1. Helpfulness (1-5): Does it address what the user asked?
2. Faithfulness (1-5): Are all claims supported by the context shown?
3. Safety (pass/fail): Is any content harmful or inappropriate?

Inter-rater reliability (IRR). When multiple reviewers score the same output, they will not always agree. IRR measures the degree of agreement. Common measures:

Cohen's Kappa (two raters, categorical labels): values above 0.6 are acceptable; above 0.8 is good
Percent agreement: simple but inflates for imbalanced classes

Low IRR signals one of three problems: the rubric is ambiguous, the task is genuinely subjective, or reviewers are not applying the rubric correctly.

Calibration. Before rating independently, reviewers should score a shared set of anchor cases together. Discuss disagreements. This step dramatically improves IRR and surfaces rubric ambiguities before they contaminate your data.

Using human feedback to improve prompts. Human ratings are valuable inputs for three improvement loops:

Rubric refinement: low-rated cases reveal failure modes. Cluster them to find systematic prompt weaknesses.
Prompt iteration: when 30%+ of reviewed cases fail a specific dimension, revisit the prompt instructions.
Golden dataset growth: reviewed cases where human rating disagrees with automated metrics are high-value additions.

Feedback signals from product. Beyond explicit annotation, implicit user signals are cheap and continuous:

Thumbs up/down ratings
Copy button clicks on responses (strong positive signal)
Follow-up questions that suggest the first answer was inadequate
Session abandonment after a response

The feedback loop:

User request -> LLM response -> User feedback (explicit or implicit)
                                       |
                             Quality metric update
                                       |
                    Flag for human review (if quality drops)
                                       |
                    Human annotates the failure case
                                       |
                    Add to golden dataset / update rubric / iterate prompt
                                       |
                              Rerun eval suite

Working example

A human feedback collection and aggregation pipeline:

from dataclasses import dataclass
from collections import defaultdict
import statistics


@dataclass
class ThumbsFeedback:
    trace_id: str
    feature_name: str
    rating: str  # 'up', 'down', 'neutral'
    timestamp_ms: int


@dataclass
class Annotation:
    trace_id: str
    feature_name: str
    rater_id: str
    helpfulness: int   # 1-5
    faithfulness: int  # 1-5
    notes: str = ''


class FeedbackAggregator:
    def __init__(self):
        self._thumbs: list[ThumbsFeedback] = []
        self._annotations: list[Annotation] = []

    def record_thumbs(self, feedback: ThumbsFeedback) -> None:
        self._thumbs.append(feedback)

    def record_annotation(self, annotation: Annotation) -> None:
        self._annotations.append(annotation)

    def satisfaction_rate(self, feature_name: str) -> float | None:
        rated = [t for t in self._thumbs
                 if t.feature_name == feature_name and t.rating in ('up', 'down')]
        if not rated:
            return None
        return sum(1 for t in rated if t.rating == 'up') / len(rated)

    def compute_irr(self, feature_name: str) -> dict:
        by_trace: dict[str, list[Annotation]] = defaultdict(list)
        for a in self._annotations:
            if a.feature_name == feature_name:
                by_trace[a.trace_id].append(a)
        multi_rated = {tid: annots for tid, annots in by_trace.items() if len(annots) > 1}
        if not multi_rated:
            return {'error': 'No multiply-rated traces found'}
        agreements = total_pairs = 0
        for annots in multi_rated.values():
            ratings = [a.helpfulness >= 3 for a in annots]
            for i in range(len(ratings)):
                for j in range(i + 1, len(ratings)):
                    total_pairs += 1
                    if ratings[i] == ratings[j]:
                        agreements += 1
        return {
            'traces_with_multiple_raters': len(multi_rated),
            'percent_agreement_helpfulness': agreements / total_pairs if total_pairs else 0.0,
        }

    def low_quality_traces(
        self, feature_name: str, threshold: float = 2.5, min_raters: int = 1
    ) -> list[str]:
        by_trace: dict[str, list[Annotation]] = defaultdict(list)
        for a in self._annotations:
            if a.feature_name == feature_name:
                by_trace[a.trace_id].append(a)
        result = []
        for trace_id, annots in by_trace.items():
            if len(annots) < min_raters:
                continue
            avg = statistics.mean((a.helpfulness + a.faithfulness) / 2 for a in annots)
            if avg < threshold:
                result.append(trace_id)
        return result

Common mistakes

Asking reviewers to rate 'overall quality' without a rubric. Reviewers will apply inconsistent mental models and IRR will be low. Always decompose into specific dimensions.
Not measuring IRR before trusting annotations. Low-IRR data is worse than no data because it gives false confidence.
No connection between annotation outcomes and prompt changes. If human reviewers identify 40 cases where the model hallucinated but that feedback never influences the prompt, the annotation effort was wasted.
Using only thumbs ratings as quality signal. Thumbs ratings are high-volume but noisy. Triangulate with more specific annotation.
Annotation fatigue. Keep sessions under 45 minutes and rotate reviewers. Quality declines significantly after the first hundred or so reviews in a session.

Try it yourself

Design a minimal annotation interface specification for a coding assistant feature. Define: the fields shown to the reviewer, the scoring rubric for three dimensions specific to code generation (correctness, readability, explanation quality), and a short calibration set of five cases with pre-agreed ratings. Then implement a compute_kappa function that takes two lists of integer ratings and returns Cohen's Kappa for two raters on a 5-point scale.