Why this matters

A single LLM API call is easy. Handling 500 concurrent users, each expecting responses in under two seconds, is a different engineering problem. The naive approach — one model process, one request at a time — falls apart immediately. This lesson covers the serving architecture decisions that let you scale LLM inference without buying ten times more hardware than you need.

Core concepts

The throughput problem

LLM inference is GPU-bound and memory-bound simultaneously. A single A100 GPU can only hold a certain number of KV-cache entries. If you serve one request at a time, you are leaving GPU compute idle during memory fetches. If you over-subscribe memory, you get OOM crashes. The art of LLM serving is maximizing utilization within memory constraints.

The key insight: batching. If five users send requests within 50 milliseconds of each other, you can process them together in a single GPU forward pass and return results to all five in roughly the same time it takes to process one. This is the fundamental lever behind all the serving frameworks.

Continuous batching

Traditional batching waits for a fixed batch to fill before processing. This adds latency when traffic is low. Continuous batching (also called dynamic batching or iteration-level scheduling) processes requests as they arrive, adding new requests to the in-flight batch at each iteration step.

This is what makes vLLM and TGI significantly more efficient than naively running a model:

Traditional:   [req1, req2, -, -, -] -> process -> [req1, req2, -, -, -] next batch
Continuous:    [req1, req2, -, -, -] -> step -> [req1, req2, req3, req4, -] next step
               (req3 and req4 joined mid-generation)

vLLM

vLLM is the most production-adopted high-throughput serving framework. It implements PagedAttention — managing KV cache as paged memory similar to OS virtual memory — which dramatically improves GPU memory utilization.

Running vLLM as a service:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3-8B-Instruct \
    --tensor-parallel-size 2 \  # split across 2 GPUs
    --max-model-len 8192 \
    --port 8000

vLLM exposes an OpenAI-compatible API. Your application code calls http://localhost:8000/v1/chat/completions — no SDK changes required when switching from the OpenAI API.

Text Generation Inference (TGI)

TGI is Hugging Face's serving framework, optimized for Hugging Face model hub compatibility and enterprise deployments. It supports tensor parallelism, quantization, and streaming.

docker run --gpus all --shm-size 1g -p 8080:80 \
    -v /models:/data \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id /data/llama-3-8b \
    --max-input-length 4096 \
    --max-total-tokens 8192

Ollama

Ollama is the right choice for local development and single-server deployments. It handles model downloading, quantization selection, and process management. For a side project or internal tool serving tens of requests per day, Ollama removes complexity.

ollama pull llama3.2
ollama serve  # starts API at localhost:11434

Call it exactly like OpenAI:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "What is 2 + 2?"}]
)

Request queuing

At moderate scale, you need a queue in front of your inference service. The queue serves three purposes:

1. Backpressure: requests pile up in the queue rather than hammering your GPU service
2. Priority routing: paying customers can jump the queue
3. Retries: failed requests can be re-enqueued without client retries

A simple Redis-backed queue with priority routing:

import asyncio
import redis.asyncio as aioredis
import json


class InferenceQueue:
    def __init__(self, redis_url: str):
        self.redis = aioredis.from_url(redis_url)
        self.high_priority_key = "inference:queue:high"
        self.normal_priority_key = "inference:queue:normal"

    async def enqueue(self, request: dict, priority: str = "normal") -> str:
        request_id = request["request_id"]
        queue_key = self.high_priority_key if priority == "high" else self.normal_priority_key
        await self.redis.rpush(queue_key, json.dumps(request))
        return request_id

    async def dequeue(self, timeout: int = 5) -> dict | None:
        # Try high priority first, then normal
        result = await self.redis.blpop(
            [self.high_priority_key, self.normal_priority_key],
            timeout=timeout
        )
        if result is None:
            return None
        _, payload = result
        return json.loads(payload)

    async def queue_depth(self) -> dict:
        high = await self.redis.llen(self.high_priority_key)
        normal = await self.redis.llen(self.normal_priority_key)
        return {"high": high, "normal": normal, "total": high + normal}

Load balancing multiple model replicas

When a single GPU is not enough, you run multiple model replicas behind a load balancer. The load balancer should be aware of replica health and queue depth, not just round-robin:

import random


class ModelReplicaPool:
    def __init__(self, replicas: list[dict]):
        # replicas: [{"url": "http://...", "weight": 1, "healthy": True}]
        self.replicas = replicas

    def select_replica(self) -> str:
        healthy = [r for r in self.replicas if r["healthy"]]
        if not healthy:
            raise RuntimeError("No healthy replicas available")
        # Weighted selection
        total_weight = sum(r["weight"] for r in healthy)
        pick = random.uniform(0, total_weight)
        running = 0
        for replica in healthy:
            running += replica["weight"]
            if pick <= running:
                return replica["url"]
        return healthy[-1]["url"]

    def mark_unhealthy(self, url: str) -> None:
        for r in self.replicas:
            if r["url"] == url:
                r["healthy"] = False
                break

Working example

A complete request router that queues work, selects a replica, calls the inference service, and handles timeouts:

import asyncio
import httpx
import logging

logger = logging.getLogger(__name__)


class InferenceRouter:
    def __init__(self, pool: ModelReplicaPool, timeout_seconds: float = 30.0):
        self.pool = pool
        self.timeout = timeout_seconds

    async def complete(self, messages: list[dict], model: str = "default") -> str:
        replica_url = self.pool.select_replica()
        try:
            async with httpx.AsyncClient(timeout=self.timeout) as client:
                response = await client.post(
                    f"{replica_url}/v1/chat/completions",
                    json={
                        "model": model,
                        "messages": messages,
                        "max_tokens": 1024,
                    }
                )
                response.raise_for_status()
                data = response.json()
                return data["choices"][0]["message"]["content"]
        except (httpx.TimeoutException, httpx.HTTPStatusError) as exc:
            logger.warning("replica_failed", extra={"url": replica_url, "error": str(exc)})
            self.pool.mark_unhealthy(replica_url)
            raise

Common mistakes

Serving one request at a time. Even without a dedicated serving framework, you can run multiple async workers. A single synchronous Flask endpoint tied to a model object will serialize every request.

Not setting KV cache memory limits. vLLM will claim all available GPU memory for KV cache by default. Set --gpu-memory-utilization 0.85 to leave headroom for OS and CUDA runtime operations.

Mixing quantization levels without benchmarking. INT4 quantization cuts memory usage by 4x but can degrade quality significantly for instruction-following tasks. Always run your evaluation suite against the quantized model before deploying.

No timeout on inference calls. A single stuck request with no timeout can hold a GPU forward pass indefinitely. Always set a timeout that is roughly 2x your P99 latency.

Ignoring batching for embedding workloads. Embedding generation scales extremely well with batching. Sending 100 texts in one request versus 100 individual requests can be 20x faster with the same hardware.

Try it yourself

1. Run Ollama locally and use the OpenAI-compatible API to call it from Python. Measure the latency difference between serial calls and asyncio.gather with five concurrent requests.
2. Implement a simple health-check polling loop that marks a replica as unhealthy after two consecutive timeouts and re-enables it after a successful probe.
3. Extend the InferenceQueue to add a result store using Redis hashes so callers can poll for their result by request_id.