AI Engineer Portal

Observability Stack for Production LLM Applications

A production LLM feature generates three distinct classes of observable events: system events (latency, errors, provider health), quality events (evaluation scores, judge results, user feedback), and business events (task completion, follow-up rates, session outcomes). Each class needs different instrumentation and different alerting logic.

The three-tier observability model

Tier 1: System health

Standard infrastructure monitoring applies here with minor additions:

Request rate and error rate — same as any API
Latency percentiles (P50, P95, P99) — LLM calls have fat-tailed latency distributions; P50 is fast, P99 is wild
Provider error classification — distinguish rate limit errors (429) from context length errors (400) from service errors (500); each has a different operational response
Token throughput — tokens per second, not just requests per second
finish_reason distribution — a spike in max_tokens is a silent quality failure, not a system failure

Tier 2: Quality health

This is where AI-specific instrumentation lives:

Sampled judge scores — faithfulness, relevance, helpfulness on 5-10% of traffic
Pass rate by feature and by category — not global; category-level granularity surfaces blind spots
Score distribution — mean and P5; a high mean with a low P5 means a subset of inputs is systematically failing
Human feedback rate — thumbs down per 100 requests; explicit negative feedback is the strongest quality signal
auto_score distribution — if your inline scorer reports uncertain scores (0.4-0.7) on more than 15% of traffic, something changed

Tier 3: Business health

Business metrics are the ultimate arbiter of quality:

Task completion rate — for task-oriented features, did users achieve their goal?
Follow-up question rate — high follow-up rate may indicate the first answer was incomplete
Copy/export action rate — users copying text signals they found it useful
Session abandonment rate after AI response — users who leave immediately after a response may have gotten a bad one

Trace schema design

Every LLM request should produce a trace record with:

{
    "trace_id": "uuid",
    "feature_name": "document_qa",
    "request_timestamp_ms": 1711900000000,
    "model": "claude-3-5-sonnet-20241022",
    "prompt_version": "qa_v3",
    "prompt_tokens": 1240,
    "completion_tokens": 312,
    "latency_ms": 1840,
    "finish_reason": "end_turn",
    "retrieval_chunks": 5,
    "top_chunk_score": 0.87,
    "auto_score": 0.81,
    "user_id_hash": "a3f7c...",  # hashed, not raw
    "session_id": "sess_...",
}

Never log raw user messages or raw responses by default. Log a truncated preview (200 chars) at INFO level, full content at DEBUG level with shorter retention.

Data pipeline from trace to alert

LLM call
    |
Trace record emitted (structured JSON to stdout)
    |
Log aggregator (Datadog, Loki, CloudWatch)
    |
Quality metrics pipeline (hourly batch)
    |
Dashboard (pass rate, score trend, cost)
    |
Alert rules (rolling window breach)
    |
On-call notification

The quality metrics pipeline is the component most teams skip. Without it, you have individual trace records but no rolling statistics. The pipeline reads traces, scores a sample with the LLM judge, computes rolling averages by feature and category, and writes the results to a metrics store.

Alert thresholds to configure before launch

Signal	Threshold	Cadence
Pass rate	< 80% rolling 4h	Hourly
Avg faithfulness	< 0.70 vs 14-day baseline	Daily
P5 score floor	< 0.30 rolling 1h	Real-time
Negative feedback rate	> 5% rolling 1h	Real-time
Provider error rate	> 2% rolling 5min	Real-time
Cost per request	> 2x 14-day avg	Daily

The most common gap

Teams build system health monitoring (Tier 1) and sometimes business monitoring (Tier 3), but skip Tier 2 entirely. The result: you get paged when the service is down, you see business metrics decline over days or weeks, but you have no data connecting the quality degradation to a specific feature, model version, or query category.

Tier 2 is the diagnostic layer. Without it, investigations start from scratch every time.

Your personal operating system for career transition.

Observability stack for production LLM applications