AI Engineer Portal
Your personal operating system for career transition.
Exercise
Implement a quality threshold gate for CI
Implement a Quality Threshold Gate for CI
An eval harness only protects your system if it blocks bad deployments. This exercise builds the gate that translates eval results into a deploy/reject decision.
What to build
Implement QualityGate:
-
Constructor — accepts a
thresholds: dictwith any subset of these keys:min_pass_rate(float, default 0.80),max_regression_count(int, default 0),min_avg_score(float, default 0.70),max_failure_rate_by_category: dict[str, float](per-category max failure rate). -
check(report: dict) -> GateResult—reportis the dict returned by aGoldenSuite.report()call. Returns aGateResultdataclass with:passed: bool,violations: list[str](human-readable strings per violated threshold),summary: str(one-sentence deploy/reject verdict). -
assert_passes(report: dict) -> None— Callscheck()and raisesRuntimeErrorwith the violations ifpassed == False. Designed for use in CI scripts.
Constraints
- Standard library only. Each violation message should name the threshold, the actual value, and the threshold value.
Evaluation / medium / Step 29 of 36
Evaluation and review loops
Separate the scoring logic from the interpretation logic. Your goal is not just a number; it is a useful next action.
- - Produces a useful signal, not decorative output
- - Makes regression review easier
- - Would support a benchmark or observability loop
- - Would this output help decide what to fix next?
- - Are important failure modes visible?
- - Does the score hide any ambiguity I should record?
Practice
Generate a variation
Generate a new exercise variation to deepen understanding or practice a related concept.
Attempt history
Recent submissions
Before you submit, decide what a strong answer should make obvious to the reviewer.
No attempts yet.