Your personal operating system for career transition.

Build an LLM-as-Judge Evaluator

Automated evaluation with a language model as the judge is one of the most important patterns in applied AI engineering. Instead of requiring human annotation for every output, you prompt a capable model with a rubric and let it score responses at scale.

What you are building

Implement a LLMJudge class that:

1. Accepts a rubric -- a plain-English description of what "good" looks like for the task.
2. Scores a response -- given a question, a generated response, and optional context, returns a score between 0.0 and 1.0 plus a one-sentence rationale.
3. Batches evaluations -- runs a list of (question, response, context) tuples and returns aggregated results.
4. Validates output -- if the judge returns malformed JSON or an out-of-range score, log a warning and return a default score of 0.5 rather than crashing.

Rubric for testing

A helpful response directly addresses the user's question, uses only information
from the provided context, and does not introduce unsupported claims.
Score 1.0 if fully helpful and faithful, 0.5 if partially, 0.0 if off-topic or hallucinated.

Constraints

• No external evaluation libraries.
• Type-hint every method.
• The judge prompt must include the rubric, question, context, and response in clearly labeled sections.
• Return a JudgeResult dataclass with: score: float, rationale: str, raw_output: str.