Your personal operating system for career transition.

Build an Evaluation-Driven Development Workflow

Evaluation-driven development means: write the eval before you write the feature, then use eval results to guide every prompt change. Build the scaffolding for this workflow.

Requirements

1. EvalCheckpoint dataclass: version (str), commit_hash (str), timestamp (datetime), pass_rate (float), avg_score (float), regression_cases (list of case_id strings), notes (str).

2. CheckpointStore class backed by a list (in-memory, no disk I/O). Methods:

   • save(checkpoint: EvalCheckpoint) -> None
   • latest() -> EvalCheckpoint | None
   • history(n: int = 10) -> list[EvalCheckpoint] — most recent n, newest first
   • is_regression(current: EvalCheckpoint, threshold: float = 0.03) -> bool — True if current pass_rate is more than threshold below the latest checkpoint's pass_rate

3. EvalGate class with method check(current: EvalCheckpoint, store: CheckpointStore) -> tuple[bool, str]. Returns (True, "OK") if:

   • No prior checkpoints exist (first run always passes)
   • Pass rate did not regress beyond threshold Returns (False, reason_string) if a regression is detected. The reason string should include the previous pass rate, current pass rate, and the list of new regression case IDs (cases in current.regression_cases not in the latest checkpoint's regression_cases).

4. A format_regression_report(gate_result: tuple[bool, str], current: EvalCheckpoint) -> str that returns a one-line summary for CI output:

   • On pass: "EVAL PASS v{version}: {pass_rate:.1%} pass rate ({avg_score:.3f} avg score)"
   • On fail: "EVAL FAIL v{version}: {reason}"