Why standard observability falls short for AI services
Web service observability answers: is it up? Is it fast? Are requests failing? These questions are necessary but not sufficient for LLM services. A service that is 100% available and sub-second can still be failing its users if the model outputs are consistently low-quality, if costs are spiraling, or if the response quality silently degraded after a model update.
Production LLM observability needs to cover three layers that traditional APM tools do not address.
Layer 1: Infrastructure observability (what you already have)
The standard signals still matter:
- Latency percentiles (P50, P95, P99) — LLM latency is highly variable. A P50 of 800ms with a P99 of 12 seconds indicates long-tail issues that users will notice.
- Error rate by status code — distinguish provider errors (5xx from the LLM API) from your application errors and rate limit responses (429).
- Queue depth — if you run a request queue, rising queue depth indicates your inference capacity is not keeping up with demand.
These come from your standard metrics stack (Prometheus, Datadog, CloudWatch).
Layer 2: LLM-specific operational metrics
These require instrumentation in your application code:
Token usage per request and per feature. Track input_tokens, output_tokens, and derived cost_usd for every LLM call. Aggregate by feature and alert when daily spend on any feature exceeds budget.
Cache hit rate. If you run a semantic or exact-match cache, track the fraction of requests served from cache. A hit rate below 15% suggests the cache is not helping much for your traffic pattern.
Provider selection and fallback rate. In a multi-provider setup, track how often each provider is used. A sudden shift from primary to secondary provider indicates the primary is degraded.
Model version distribution. When running A/B tests or canary deployments, track the fraction of requests going to each model version. This confirms the routing is behaving as configured.
Layer 3: Quality observability
This layer is unique to AI services and the hardest to build:
Sampled quality scoring. Run a lightweight quality check on a random sample of responses (e.g., 1-5% of traffic). Use a cheap classifier or LLM judge to score each response. Alert when the rolling average quality score drops below a threshold.
Output anomaly detection. Track response length distribution, refusal rate (responses containing "I cannot" or similar patterns), and empty/truncated response rate. Anomalies in these signals often precede quality complaints.
User feedback signal. If your product has thumbs up/down or explicit feedback, wire this into your observability stack. Even a 0.1% feedback response rate provides a meaningful quality signal.
The trace as the fundamental unit
Every production LLM request should produce a trace with spans covering:
- Request received + routing decision
- Cache lookup (hit or miss)
- Context retrieval (if RAG) + retrieved chunk count and top score
- Prompt construction + token count
- LLM call + model, input tokens, output tokens, latency
- Response post-processing + any validation failures
A trace that captures all of these makes it possible to answer "why did this specific request produce a bad response?" in under five minutes.
Practical alert thresholds
| Signal | Alert condition |
|---|---|
| P95 LLM latency | > 8 seconds (5-minute window) |
| Provider error rate | > 2% (5-minute window) |
| Cache hit rate | < 10% (1-hour window) |
| Quality score (sampled) | < 0.80 rolling 100 requests |
| Daily feature cost | > 120% of budget |
| Fallback rate | > 5% (1-hour window) |
Start with loose thresholds and tighten them as you understand your normal operating range.
What NOT to alert on
Resist the temptation to alert on every metric. Alert fatigue is the enemy of operational discipline. Only alert when a metric indicates a user-impacting problem that requires human action in the next 30 minutes. Model latency trending up over a week is a planning item, not a 3 AM page.