Your personal operating system for career transition.

Build a Multi-Metric Eval Pipeline

Production evaluation rarely relies on a single metric. A useful pipeline runs several scorers in parallel, aggregates their outputs into a composite, and flags cases that fail any individual dimension.

What to build

Implement MultiMetricPipeline:

1. Constructor — accepts scorers: dict[str, Callable[[str, str], float]] mapping metric name to scorer function. Optionally accepts weights: dict[str, float] (default: equal weight).

2. run(cases: list[dict]) -> list[dict] — Each case has "input", "response", and "reference". Returns per-case results: {"case_id", "scores": {metric: float}, "composite_score", "passed_all", "failed_metrics"}.

3. aggregate(results: list[dict]) -> dict — Returns {"n", "avg_composite", "per_metric_avg": {metric: float}, "cases_failing_any": int}.

4. worst_cases(results, n=5) -> list[dict] — Top n cases by lowest composite score, sorted ascending.

Constraints

• Standard library only. If a scorer raises, record score 0.0 for that metric and continue.
• Composite score = weighted average of individual metric scores.
• passed_all = True only when every metric score >= 0.7.