Your personal operating system for career transition.

Implement a Sampling Strategy for Production Evaluation

You cannot run an expensive LLM judge on every production request. A good sampling strategy applies the expensive scorer to the right requests — not random ones.

What to build

Implement EvalSampler:

1. should_sample_random(sample_rate: float) -> bool — Returns True with probability sample_rate. Use random.random().

2. should_sample_rule_based(event: dict) -> bool — Returns True if ANY of these apply: event has "user_feedback": "negative", event has "finish_reason": "max_tokens", event has "auto_score" and that score is between 0.4 and 0.7 (the uncertain zone), event has "is_new_query_type": True.

3. classify(event: dict, sample_rate: float = 0.05) -> dict — Returns {"sampled": bool, "reason": str}. Reason values: "rule_triggered", "random_sample", "not_sampled". Rule-based takes priority over random.

4. batch_classify(events: list[dict], sample_rate: float = 0.05) -> dict — Returns {"total", "sampled", "by_reason": dict, "sample_rate_actual": float}.

Constraints

• Standard library only. Make should_sample_random deterministic when called with a fixed random.seed.