Production RAG: Operational Checklist
Most RAG tutorials end with a working demo. Production RAG requires eight additional concerns that are not discussed in tutorials because they are not interesting to write about — but they determine whether the system keeps working after launch.
1. Query caching
Identical or near-identical queries should not hit the embedding model and vector DB every time. Cache by query hash with a TTL appropriate for your content freshness requirements. Help-center bots typically see 20–40% of queries repeat within an hour — caching eliminates that retrieval and generation cost entirely.
Caching also significantly improves latency for the repeated query case. Users who ask common questions get near-instant responses instead of waiting for the full pipeline.
2. Retrieval and generation fallbacks
Define fallback strategies before launch, not after an outage:
- Primary retriever fails → fall back to a secondary index or a simpler keyword-only retrieval.
- Primary LLM fails → fall back to a cheaper, faster model with a more conservative prompt.
- Both fail → return a graceful "I'm unable to answer right now" message instead of a 500 error.
Use a circuit breaker on the primary retriever so consecutive failures stop adding timeout latency to every request.
3. Latency monitoring by pipeline step
Log latency separately for embedding, retrieval, reranking, and generation. Do not just measure total latency. When a user complains about slowness, you need to know which step caused it. P50 and P95 per step tell you whether the problem is consistent or spiky.
4. Cost visibility per query
Calculate cost per query from day one: embedding tokens, input context tokens, output tokens, and any reranking API calls. At low volume this is negligible; at scale it determines your unit economics. Attribute cost by feature (not all RAG queries are the same product feature) so you can see which features are expensive before a traffic spike surprises you.
5. Incremental indexing
Do not re-index the entire corpus on every deploy. Track document content hashes. Only re-embed documents that have changed. Delete old vectors by document ID when content changes. This reduces indexing cost and prevents stale content from sitting in the index.
6. Access control enforcement at retrieval time
If your corpus contains documents with different access levels, enforce access control at retrieval — not at the application layer after retrieval. A user who should not see document X should not receive it as a retrieved chunk even if the model does not quote it directly. The only safe place to enforce this is the metadata filter applied before semantic scoring.
7. Evaluation regression tracking
Run your Recall@K test set and faithfulness eval on every significant deployment. A change to chunking, embedding model, or retrieval configuration should be validated against your eval suite before reaching production. Without this, quality regressions are invisible until users complain.
8. Content freshness rules
Decide what "fresh enough" means for your corpus. Documents that expire should be deleted from the index, not just deprioritized. Stale content in the index produces confidently wrong answers — the most damaging failure mode for user trust.
Practical takeaway
Go through this list before any RAG feature ships to production. Most of these concerns are low implementation cost but high operational impact. The ones you skip will be the ones that cause your first incident.