方寸 Portal

AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Implement a quality threshold gate for CI

Implement a Quality Threshold Gate for CI

An eval harness only protects your system if it blocks bad deployments. This exercise builds the gate that translates eval results into a deploy/reject decision.

What to build

Implement QualityGate:

  1. Constructor — accepts a thresholds: dict with any subset of these keys: min_pass_rate (float, default 0.80), max_regression_count (int, default 0), min_avg_score (float, default 0.70), max_failure_rate_by_category: dict[str, float] (per-category max failure rate).

  2. check(report: dict) -> GateResultreport is the dict returned by a GoldenSuite.report() call. Returns a GateResult dataclass with: passed: bool, violations: list[str] (human-readable strings per violated threshold), summary: str (one-sentence deploy/reject verdict).

  3. assert_passes(report: dict) -> None — Calls check() and raises RuntimeError with the violations if passed == False. Designed for use in CI scripts.

Constraints

  • Standard library only. Each violation message should name the threshold, the actual value, and the threshold value.

Evaluation / medium / Step 29 of 36

Practice stage

Evaluation and review loops

Hint

Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.

Success criteria
  • - Produces a useful signal, not decorative output
  • - Makes regression review easier
  • - Would support a benchmark or observability loop
Review checklist
  • - Would this output help decide what to fix next?
  • - Are important failure modes visible?
  • - Does the score hide any ambiguity I should record?

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.