方寸 Portal

AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Create an eval gate for CI/CD pipelines

Implement a EvalGate class that can be used as a quality gate in a CI/CD pipeline:

  1. run(cases: list[dict]) -> EvalGateResult — evaluate all cases and return a result object
  2. Each case has: id: str, input: str, expected: str, actual: str
  3. A case passes if the actual output contains all key terms from the expected output (use a simple term overlap check: at least 60% of lowercased words in expected that are longer than 3 characters must appear in actual)
  4. EvalGateResult should have: passed: bool, pass_rate: float, failed_cases: list[str] (list of case IDs), total: int
  5. EvalGate takes a threshold: float (default 0.85) in its constructor — passed is True if pass_rate >= threshold
  6. Add a to_ci_report() -> str method on EvalGateResult that returns a human-readable summary suitable for CI log output

Write a test that demonstrates a gate passing with 9/10 cases and blocking with 8/10 cases (threshold=0.85).

Deployment / medium / Step 4 of 6

Practice stage

General drill

Hint

Keep the solution explicit and reviewable.

Success criteria

Make the solution explicit, debuggable, and easy to explain.

Review checklist

Review where the boundary is, what gets validated, and what would be hard to debug later.

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.