The demo-to-production gap
An LLM demo that works 100% of the time in a controlled environment will encounter conditions in production that the demo never hit: users with unusual input patterns, provider outages at 2 AM, model updates that change output format, token budget exhaustion, and cost spikes from unexpected traffic.
This gap is not about model capability. It is about engineering discipline applied to an AI-specific deployment surface.
Decision 1: Synchronous or asynchronous response
For requests that take 2-20 seconds, you have two choices:
Synchronous (HTTP request-response): The client waits with a long-polling connection. Simple to implement. Works well when P99 latency is under 15 seconds and users expect real-time feedback. Stream the response with SSE (Server-Sent Events) to improve perceived performance.
Asynchronous (queue-based): The client submits a job and gets a job_id. It polls or subscribes to a result channel. Better when latency is unpredictable or when you need to buffer demand during traffic spikes. Required when processing involves multiple LLM calls chained together (agents, multi-step pipelines).
For most web product features, start synchronous with streaming. Move to async when P99 latency exceeds user tolerance or when you need to decouple ingestion from inference capacity.
Decision 2: Deployment unit for the AI feature
Monolith extension: Add LLM calls directly to your existing service. Fastest path to production. Fine when LLM calls are a small fraction of traffic. Risk: a provider outage or latency spike can degrade your entire service.
Sidecar LLM service: A separate microservice handles all LLM calls. The main product calls it over HTTP. Isolates LLM-specific concerns (rate limiting, caching, cost tracking, circuit breakers). Adds network hop complexity but contains failure blast radius.
Serverless function: Deploy LLM features as serverless functions with per-invocation billing. Eliminates idle cost. Cold start latency (3-8 seconds for Python + large deps) makes this a bad fit for interactive features. Better for batch processing and scheduled jobs.
For new features in an existing product, the sidecar approach is often the right call — it lets you deploy LLM-specific reliability patterns without mixing them into your main service's codebase.
Decision 3: Prompt and model version management
The most important operational pattern most teams skip: treat prompts as versioned artifacts.
prompts/
summarize_v1.txt ← previous production version
summarize_v2.txt ← current production version
summarize_v3.txt ← candidate being tested
Load the active version from configuration:
version = os.environ.get("PROMPT_VERSION_SUMMARIZE", "v2")
This gives you:
- Version history via git diff
- Instant rollback without a code deployment
- Canary testing by routing some users to v3 before full promotion
Without this, a prompt change that breaks production has no clean rollback path.
Decision 4: Rollout strategy for AI features
Hard launch (100% at once): Only for features with comprehensive eval coverage and where quality regression is easily detected. High risk.
Feature flag rollout: Deploy the feature off. Enable for internal users first. Expand to 5%, 20%, 50%, 100% while monitoring quality signals. Most common pattern for AI features.
Canary by model version: Keep the current model version serving 90% of traffic. Route 10% to the new version. Compare quality metrics before promoting.
Shadow mode: Run the new version in parallel with the current one, log both outputs, but only return the current version's response to users. Use shadow mode to evaluate a new model or prompt against real traffic before exposing it.
Decision 5: Cost controls before launch
Before any AI feature goes to production:
- Token budget enforcement — cap prompt size at construction time. Never let user input flow directly into a prompt without a size check.
- Per-user daily limits — rate limit how many LLM calls a single user can make per day.
- Cost monitoring — instrument every LLM call with feature label and model. Generate daily cost reports grouped by feature.
- Budget alerts — configure an alert in your provider billing console at 80% of your monthly budget. Configure an in-application alert when daily feature cost exceeds a threshold.
Cost controls are much easier to add before a feature launches than after users have formed expectations about what the feature does.
The minimum viable production checklist
Before shipping an AI feature to production, verify:
- Provider API key stored in a secret manager, not in code or environment variables
- Retry with exponential backoff on transient provider errors
- Timeout on every LLM call (max 30-60 seconds)
- Circuit breaker or fallback when the provider is down
- Token budget enforcement on prompt construction
- Cost event logged for every LLM call
- Health check endpoint that probes provider reachability
- Eval gate in CI (even a simple 20-case golden test suite)
- At least one graceful degradation path