Why standard tracing falls short for LLM applications
Standard application tracing records HTTP request/response cycles. An LLM application that takes 3 seconds to respond cannot be debugged from HTTP-level traces alone — you cannot tell whether the time was spent on retrieval, embedding lookup, prompt assembly, the model call, or response parsing.
Span-level tracing breaks the pipeline into individual operations with their own latency, status, and metadata. This converts "something was slow" into "the model call was 2.8 seconds; retrieval was 80ms."
What to instrument
The pipeline root span
Every user-facing request should have a root span that covers the entire operation:
with tracer.start_as_current_span("rag_request") as span:
span.set_attribute("user_id", user_id)
span.set_attribute("feature", "chat")
span.set_attribute("session_id", session_id)
Retrieval span
with tracer.start_as_current_span("retrieval") as span:
span.set_attribute("query_length_chars", len(query))
span.set_attribute("top_k", top_k)
results = vector_store.search(query_embedding, top_k=top_k)
span.set_attribute("results_returned", len(results))
span.set_attribute("top_score", results[0].score if results else 0.0)
The top_score attribute is especially useful: a score below 0.5 on the top result often indicates retrieval failure even when the number of results looks correct.
LLM call span
with tracer.start_as_current_span("llm_call") as span:
span.set_attribute("model", model)
span.set_attribute("input_tokens", input_tokens)
response = client.chat(messages)
span.set_attribute("output_tokens", response.usage.completion_tokens)
span.set_attribute("finish_reason", response.choices[0].finish_reason)
span.set_attribute("cost_usd", compute_cost(model, input_tokens, output_tokens))
The finish_reason attribute catches truncated responses (length) and content filter triggers (content_filter) — both are silent failures that look like normal responses in HTTP-level monitoring.
Attributes that make traces actionable
| Attribute | Why it matters |
|---|---|
model | Compare latency and quality across model versions |
input_tokens | Correlate with latency and cost |
output_tokens | Detect truncation |
finish_reason | Catch silent failures |
retrieval.top_score | Detect retrieval failure |
cache_hit | Measure cache effectiveness |
provider_name | Track multi-provider routing |
cost_usd | Per-request cost tracking |
Connecting traces to quality
Standard observability traces give you latency and errors. For LLM applications, add quality signals as span attributes:
span.set_attribute("quality_score", judge_result.score)
span.set_attribute("quality_passed", judge_result.score >= 0.7)
span.set_attribute("faithfulness", faithfulness_score)
This lets you filter traces by quality: "show me all traces where quality_passed=False in the last hour." Without quality in traces, you debug slow requests but not poor-quality requests.
Export targets
OpenTelemetry supports multiple backends without changing application code:
- Jaeger (open source): good for local development and self-hosted deployments
- Honeycomb: excellent for column-store querying of trace attributes
- Datadog APM: good if already using Datadog for infrastructure monitoring
- LangSmith / Langfuse: LLM-specific tracing with built-in quality tracking
The OTLP exporter is the standard: configure the endpoint and all OTLP-compatible backends accept the same traces.
Practical setup
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://collector:4317")))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("my-llm-app")
The BatchSpanProcessor buffers spans and exports them asynchronously. The synchronous SimpleSpanProcessor is only appropriate for development and testing.