Your personal operating system for career transition.

Why request-count limits fail for LLM services

Standard API rate limiting counts requests per minute. This model breaks immediately for LLM workloads because request cost variance is extreme: a single 100,000-token prompt costs as much as 100 typical requests. A user who learns they can send one massive request per minute will do exactly that.

The correct model enforces two independent limits simultaneously: requests per minute AND tokens per minute.

The two-dimension limit model

User quota example:
  Free tier:    60 requests/min  AND  50,000 tokens/min
  Pro tier:    120 requests/min  AND 200,000 tokens/min
  Enterprise:  unlimited req      AND custom token budget

A request is rejected if either limit is exceeded. The token estimate is consumed optimistically before the call — if you wait for actual token counts after the call, the limit is not enforced until after the cost has been incurred.

Sliding window vs fixed bucket

A fixed-bucket counter resets every 60 seconds. A user who learns the reset schedule can make two full-limit bursts within seconds of a bucket boundary. The sliding window evaluates the most recent 60-second period regardless of when the clock ticks.

For the implementation, a per-user deque of (timestamp, tokens) entries is evicted on each check. Sum the tokens in the remaining entries to determine current usage.

Multi-tier user quotas

Rate limits should be per-user, not global. A single user hitting limits should not block other users. In a multi-tenant service, implement at three levels:

• Per-user: primary defense against individual abuse
• Per-feature: prevent one feature from consuming all capacity (useful for internal services)
• Global: emergency circuit breaker if total service load spikes

Communicating limits to callers

Follow HTTP conventions:

• 429 Too Many Requests on limit exceeded
• X-RateLimit-Limit-Requests: 60 in every response
• X-RateLimit-Remaining-Tokens: 47,250 in every response
• Retry-After: 14 when rate-limited (seconds until the oldest window entry expires)

The Retry-After header is the most important. Without it, clients that receive a 429 either retry immediately (compounding the problem) or use an arbitrary backoff that wastes quota.

Practical considerations

Token estimation vs actual counts. Optimistic estimation is necessary because you do not know actual token counts until after the call. Use an approximation (4 characters per token) for enforcement, then record actual usage post-call for reporting and billing.

Async and multi-instance deployment. Per-instance in-memory rate limiters allow N× the intended limit when N service instances run. Production rate limiters must use Redis to maintain shared state across instances. Redis ZADD/ZRANGEBYSCORE/ZREMRANGEBYSCORE operations implement a sliding window atomically.

Burst allowance. Strict per-minute enforcement penalizes legitimate use cases (batch jobs, automated tests). Consider a burst multiplier: allow up to 2× the per-minute limit in a 10-second window before enforcing the longer-term average.