Your personal operating system for career transition.

Implement Token-Level Faithfulness Scoring

Faithfulness measures whether a generated answer is supported by the retrieved context. A simple approach checks whether sentences from the answer appear verbatim. A better approach uses token-level attribution.

What to build

Implement FaithfulnessScorer:

1. score(answer: str, context_chunks: list[str]) -> FaithfulnessResult — Returns a FaithfulnessResult with: score (float 0.0-1.0), attributed_tokens (tokens in answer found in any context chunk), total_tokens, coverage (attributed/total), unsupported_ngrams (list of 3-grams from the answer not found in any context chunk).

2. score_batch(cases: list[dict]) -> dict — Each case has "answer" and "context_chunks". Returns {"avg_score", "avg_coverage", "n", "low_faithfulness_cases": [index]} where low_faithfulness_cases contains indices of cases scoring below 0.5.

Token approach

• Tokenize by lowercasing and splitting on whitespace and punctuation.
• An answer token is "attributed" if it appears in any context chunk (after tokenizing the chunk the same way).
• Unsupported 3-grams: consecutive 3-token sequences in the answer where none of the three tokens appear in any context token set.

Constraints

• Standard library only. Empty answer or empty context: return score=0.0.