What semantic caching is
Exact-match caching serves the same stored response when the input hash is identical. Semantic caching serves a stored response when the input is semantically equivalent — "What is your return policy?" and "How do I return a product?" should return the same cached answer.
The mechanism: embed each query with an embedding model. On a new query, compare its embedding to stored embeddings using cosine similarity. If the best match exceeds a threshold (e.g., 0.92), return the cached response.
When semantic caching helps
Semantic caching delivers significant value in specific workload shapes:
FAQ and support features. A support bot for a SaaS product handles thousands of queries per day, but only a few hundred distinct question types. Caching the most common question clusters achieves 30-60% hit rates.
Documentation lookups. "How do I configure X?" questions cluster tightly in embedding space. Hit rates of 40-70% are achievable.
Repetitive analysis tasks. If users submit similar documents or run similar reports, embeddings of the input queries will cluster and benefit from caching.
Semantic caching does NOT help for:
- Creative generation (different users want different creative content)
- Personalized responses (user A and user B should get different answers)
- Real-time data queries (anything where staleness matters)
- Highly variable queries where no two inputs are semantically close
The threshold calibration problem
The similarity threshold is the most important tuning parameter. It controls the tradeoff between:
- High threshold (e.g., 0.97): Low hit rate, high response accuracy. Only serves cached results when queries are nearly identical.
- Low threshold (e.g., 0.80): High hit rate, risk of serving wrong cached answer. "How do I cancel my subscription?" might incorrectly match "How do I pause my subscription?" at low thresholds.
How to calibrate
-
Collect real traffic samples. Sample 500-1000 real query pairs from your production logs.
-
Manually label pairs as equivalent or different. "What is your return policy?" and "How do I return a purchase?" are equivalent. "What is your return policy?" and "What is your privacy policy?" are different despite some lexical similarity.
-
Compute cosine similarity for each pair. Plot the distribution of similarities for equivalent pairs and different pairs.
-
Find the threshold that maximizes precision at acceptable recall. You want the threshold where most equivalent pairs are above it and most different pairs are below it.
For general English text with high-quality embeddings (text-embedding-3-small or similar), the separation typically occurs around 0.90-0.95. For domain-specific text (medical, legal, code), the distribution shifts and you may need a different threshold.
- Monitor hit rate and false positive rate in production. Track what fraction of cache hits were "correct" (user was satisfied with the cached response). A false positive rate above 5% suggests the threshold is too low.
Cache invalidation
Semantic caches need expiration policies. A cached answer about your pricing from three months ago may now be wrong. Options:
- TTL-based expiration — simplest; expire entries after N days. Set TTL based on how often the underlying information changes.
- Document-update invalidation — when a source document changes (FAQ updated, policy changed), invalidate all cache entries whose cached queries are semantically similar to the changed document.
- Manual invalidation — maintain an admin endpoint to flush cache entries by topic or keyword. Useful when you make a major product change and want to clear related cached answers.
Cost math: is it worth it?
At $0.00015/1k tokens (gpt-4o-mini) with average 500 input tokens and 200 output tokens per request:
- Cost per request without cache: ~$0.00021
- At 40% hit rate with 10,000 daily requests: saves ~4,000 LLM calls/day = ~$0.84/day = ~$25/month
For a feature with gpt-4o at $0.0025/$0.01 per 1k tokens:
- Cost per request: ~$0.003
- Same 40% hit rate at 10,000 requests/day: saves ~$12/day = ~$360/month
The math favors caching more aggressively as model tier increases. For expensive models on repetitive workloads, semantic caching often pays for the infrastructure cost many times over.
Implementation note on embedding cost
Embedding every query has a cost (~$0.00002 per query with text-embedding-3-small at 200 tokens). For 10,000 daily queries, that is $0.20/day — well worth it if the cache hit rate saves even 10 LLM calls.
Store embeddings with queries rather than recomputing them. Recompute only when you upgrade embedding models (which also invalidates the cached similarity scores).