Useful evaluation metrics
Separate retrieval quality, answer faithfulness, latency, cost, and user task success so metrics guide your next engineering move.
The test for a useful metric is simple:
If this gets worse, do I know what kind of investigation or fix to start with?
If not, the metric may be decorative rather than operational.
The categories that usually matter
Retrieval quality
Ask whether the system found the right evidence.
Answer faithfulness
Ask whether the answer stayed supported by the evidence it was given.
Latency
Ask whether the feature remains usable in the product context.
Cost
Ask whether the architecture is sustainable under real usage.
User-task usefulness
Ask whether the feature helped the user complete the intended job.
Why one score fails
A single composite score hides too much:
- a system can be fast and wrong
- accurate but too slow
- grounded but unhelpful
Breaking metrics apart gives you engineering leverage.
What to do with a metric
Every metric should support one of these actions:
- investigate
- compare versions
- trigger review
- prioritize work
If it cannot do any of those, question whether it belongs on the dashboard.
Good metric design habits
- keep benchmark sets small and trusted at first
- annotate important edge cases
- pair scores with notes when qualitative failure matters
- review regressions on a schedule
Practical takeaway
Use metrics that narrow the search space of failure instead of creating the illusion of certainty.