Cost-Aware Evaluation Strategies
The naive evaluation approach — run GPT-4o as a judge on every response — is financially unsustainable at production scale. A system generating 100,000 responses per day with a 500-token judge prompt and 200-token response costs roughly $150-200/day in judge fees alone. Cost-aware evaluation is not about cutting corners. It is about spending your evaluation budget where it produces the most signal.
The cost hierarchy
Different evaluation methods differ by orders of magnitude in cost:
| Method | Relative cost | Throughput | Best for |
|---|---|---|---|
| Exact string match | ~0 | Unlimited | Structured output, classification |
| Schema/type validation | ~0 | Unlimited | JSON extraction, tool calls |
| Regex / keyword check | ~0 | Unlimited | Format compliance, required terms |
| Embedding similarity | Low | High | Semantic match, paraphrase detection |
| Small judge model (Haiku, GPT-4o Mini) | Medium | High | Continuous production sampling |
| Large judge model (Sonnet, GPT-4o) | High | Medium | Calibration, golden set evaluation |
| Human annotation | Very high | Low | Ground truth, rubric calibration |
The tiered evaluation architecture
A practical production system uses all tiers simultaneously, routing each evaluation decision to the cheapest tier that can resolve it.
Tier 1: Always-on deterministic checks. Run on 100% of responses. Validate output schema, check required fields, detect truncation (finish_reason = max_tokens), flag empty responses. These catch hard failures instantly with no LLM cost.
Tier 2: Sampled small-model judge. Run on 5-10% of production responses, continuously. A small model (GPT-4o Mini, Claude Haiku) scores faithfulness and relevance. Use the score distribution to maintain a real-time quality dashboard. When the mean score drops or the distribution shifts, trigger a deeper investigation.
Tier 3: Full golden set with large model. Run on every deploy (not continuously). A large judge model evaluates the full golden dataset. This is the regression gate — it costs $5-20 per run but only runs a few times per day.
Tier 4: Human annotation batch. Run monthly or when judge calibration degrades. Label 50-100 cases per quarter to recalibrate judges and refresh the golden dataset. This is the most expensive tier but the ground truth source.
Rule-based sampling for tier 2
Naive random sampling misses the important cases. A better approach: sample by signal strength.
Always send to judge (100% sampling):
- Negative user feedback (thumbs down, explicit "wrong")
- Auto-scorer uncertainty zone (score 0.35-0.65)
finish_reason == max_tokens(output truncated)- Error or retry in the LLM call
- First 100 responses from a new feature or prompt version
Apply random sampling (5-10%):
- Normal production traffic with confident auto-scorer score
Never judge (0%):
- Responses with schema validation failures (already flagged as hard fail)
- Duplicate (question, response) pairs seen in the last 24 hours
Caching judge results
Judge results for identical (question, response, context_hash) triples can be cached. In practice, you often evaluate the same golden set cases repeatedly as you iterate on prompts. Cache the judge output for the (case_id, response_hash) pair. When the response changes, invalidate the cache for that case.
Cost estimation as a first-class metric
Track judge spend as a feature of the evaluation system, not as an afterthought:
- Total judge cost per deploy run
- Cost per 1,000 production responses evaluated
- Judge cost as a percentage of generation cost
When judge cost exceeds 20% of generation cost, your sampling rate or judge model selection is misconfigured. The rule of thumb: judge spend should be 1-5% of total LLM spend.
What you cannot cut
Some evaluations should not be sampled:
- Safety evaluation (toxicity, PII detection) on user-visible outputs — run on 100%
- Evaluations that gate a regulatory compliance requirement — run on 100% or sample at the highest rate your budget allows
- Any evaluation that you plan to present to leadership as a quality metric — run on enough volume that the confidence interval is meaningful