Request lifecycle of an LLM feature

Why this matters

When an LLM feature behaves badly in production — slow responses, strange outputs, occasional failures — you need to know where in the request pipeline the problem lives. Is it the context assembly? The provider call? The response parsing? The persistence layer? Engineers who understand the full request lifecycle debug in minutes. Engineers who see the feature as a black box debug for hours.

The lifecycle also matters for cost. Every stage has a cost profile, and the engineers who track it are the ones who can answer "why did our AI costs go up 40% this month?" with something better than a shrug.

Core concepts

The five-stage lifecycle:

1. User input validation — sanitize, check for injection patterns, enforce length limits before touching the LLM
2. Context assembly — load user state, retrieve relevant data, count tokens, trim to budget
3. Provider API call — send the messages array, handle streaming or blocking response, track latency
4. Response parsing — extract structured data, validate format, handle malformed output
5. Persistence and logging — write the response, log tokens/latency/cost, update conversation history

Each stage can fail independently. Good architecture makes failures at each stage visible.

Token counting and cost tracking. Every provider charges per token. The cost formula is:

cost = (input_tokens × input_price) + (output_tokens × output_price)

As of early 2026 approximate rates:

• Claude 3.5 Haiku: ~$0.80 / 1M input, $4.00 / 1M output
• Claude 3.7 Sonnet: ~$3.00 / 1M input, $15.00 / 1M output
• GPT-4o mini: ~$0.15 / 1M input, $0.60 / 1M output
• GPT-4o: ~$2.50 / 1M input, $10.00 / 1M output

Track input and output tokens separately — a feature with short inputs but verbose outputs has a very different cost profile than one with long context and short answers.

Retry logic. Provider APIs fail. The correct retry strategy differs by failure type:

• 429 RateLimitError: retry with exponential backoff (start at 1s, cap at 60s)
• 500/503 ServerError: retry 2–3 times with short backoff
• 400 BadRequestError: do NOT retry — fix the request
• timeout: retry once with a slightly longer timeout
• AuthenticationError: do NOT retry — alert on-call

Fallback strategies. When retries are exhausted:

• Fall back to a cheaper/smaller model
• Return a cached or static response
• Degrade gracefully with a "I couldn't complete that" message
• Queue for async processing if real-time is not required

Working example

Here is a complete request lifecycle for a customer support assistant:

import time
import logging
from dataclasses import dataclass, field
from typing import Optional
import anthropic

logger = logging.getLogger(__name__)
client = anthropic.Anthropic()

@dataclass
class RequestTrace:
    request_id: str
    input_tokens: int = 0
    output_tokens: int = 0
    latency_ms: int = 0
    model: str = ""
    retry_count: int = 0
    error: Optional[str] = None
    cost_usd: float = 0.0

PRICING = {
    "claude-haiku-4-5": {"input": 0.80e-6, "output": 4.00e-6},
    "claude-sonnet-4-5": {"input": 3.00e-6, "output": 15.00e-6},
}

def compute_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    rates = PRICING.get(model, {"input": 3.00e-6, "output": 15.00e-6})
    return (input_tokens * rates["input"]) + (output_tokens * rates["output"])

def call_with_retry(
    messages: list[dict],
    model: str = "claude-haiku-4-5",
    max_retries: int = 3,
    timeout: float = 30.0,
) -> tuple[anthropic.types.Message, int]:
    """Call the LLM with retry logic. Returns (response, retry_count)."""
    retryable_codes = {429, 500, 502, 503, 529}
    last_error = None

    for attempt in range(max_retries + 1):
        try:
            response = client.messages.create(
                model=model,
                max_tokens=1024,
                messages=messages,
                timeout=timeout,
            )
            return response, attempt
        except anthropic.RateLimitError as e:
            last_error = e
            wait = min(2 ** attempt, 60)
            logger.warning(f"Rate limit hit, waiting {wait}s (attempt {attempt + 1})")
            time.sleep(wait)
        except anthropic.APIStatusError as e:
            if e.status_code in retryable_codes:
                last_error = e
                time.sleep(1.5 ** attempt)
                continue
            raise  # non-retryable, propagate immediately
        except anthropic.APITimeoutError as e:
            last_error = e
            timeout *= 1.5  # give it more time on retry
            if attempt < max_retries:
                continue
            raise

    raise last_error or RuntimeError("Max retries exceeded")

def handle_support_request(
    user_input: str,
    user_id: str,
    conversation_history: list[dict],
) -> tuple[str, RequestTrace]:
    """
    Full request lifecycle: validate → assemble → call → parse → log.
    Returns (response_text, trace).
    """
    import uuid
    trace = RequestTrace(request_id=str(uuid.uuid4()))

    # Stage 1: Input validation
    if len(user_input) > 4000:
        user_input = user_input[:4000]  # truncate, don't reject
    if not user_input.strip():
        return "Please provide a question I can help with.", trace

    # Stage 2: Context assembly
    messages = [
        {
            "role": "system",
            "content": (
                "You are a helpful customer support agent. Be concise and helpful. "
                "If you do not know something, say so clearly."
            ),
        },
        *conversation_history[-6:],  # last 3 turns
        {"role": "user", "content": user_input},
    ]

    # Stage 3: Provider API call with timing
    model = "claude-haiku-4-5"
    start_ms = time.monotonic() * 1000
    try:
        response, retry_count = call_with_retry(messages, model=model)
    except Exception as e:
        trace.error = str(e)
        logger.error(f"[{trace.request_id}] Provider call failed: {e}")
        return "I'm having trouble processing your request right now.", trace
    trace.latency_ms = int(time.monotonic() * 1000 - start_ms)
    trace.retry_count = retry_count

    # Stage 4: Response parsing
    response_text = response.content[0].text.strip()

    # Stage 5: Persistence and logging
    trace.model = model
    trace.input_tokens = response.usage.input_tokens
    trace.output_tokens = response.usage.output_tokens
    trace.cost_usd = compute_cost(model, trace.input_tokens, trace.output_tokens)

    logger.info(
        f"[{trace.request_id}] "
        f"model={model} "
        f"tokens={trace.input_tokens}+{trace.output_tokens} "
        f"cost=${trace.cost_usd:.6f} "
        f"latency={trace.latency_ms}ms "
        f"retries={trace.retry_count}"
    )

    return response_text, trace

Notice what the trace captures: every piece of information you need to debug a production issue. Request ID, token counts, cost, latency, retry count, and error details. This is the operational data that makes AI features debuggable.

Common mistakes

1. Not logging request IDs. When a user reports a bad response, you need to find the exact request in your logs. Generate a UUID per request and thread it through every log line.

2. Retrying non-retryable errors. A 400 BadRequestError means your payload was wrong. Retrying it 3 times wastes 3x the API calls and delays the error report. Only retry transient errors (5xx, 429).

3. No timeout on provider calls. Without a timeout, a slow provider call will block a thread indefinitely. Set aggressive-but-reasonable timeouts (15–30s for most use cases) and handle the timeout exception explicitly.

4. Parsing response text with assumptions. response.choices[0].message.content can be None if the model stopped for a content policy reason. Always handle the null case.

5. Swallowing costs. Logging token counts but not computing dollars means you will not notice a cost regression until the billing invoice arrives.

Try it yourself

Extend handle_support_request to support a fallback model. If the primary model (claude-sonnet-4-5) fails or exceeds a latency budget (say, 5 seconds), fall back to claude-haiku-4-5. Log whether a fallback was used and include it in the trace. Think about whether you want to fall back on every slow response or only when the primary call actually fails.