Your personal operating system for career transition.

Compute Semantic Similarity for Evaluation

Embedding-based similarity captures whether two texts mean the same thing even with different wording — essential for evaluating open-ended generation where BLEU/ROUGE produce false negatives.

What to build

Implement SemanticSimilarityScorer:

1. score(prediction, reference) -> float — Embeds both with injected embed_fn, returns cosine similarity (0.0-1.0). Returns 0.0 (with a logged warning) if embed_fn raises.

2. score_batch(cases: list[dict]) -> dict — Each case has "prediction" and "reference". Returns {"avg_similarity", "above_threshold", "n"}. Threshold set in constructor (default 0.8).

3. find_low_similarity_cases(cases, threshold=None) -> list[dict] — Returns cases below threshold with "similarity" added.

Constraints

• Standard library only (implement cosine manually). No external embedding library.