AI Engineer Portal
Your personal operating system for career transition.
Exercise
Implement token-based rate limiting for an LLM API
Implement Token-Based Rate Limiting for an LLM API
Request-count rate limiting is insufficient for LLM services. A user sending one 100,000-token request consumes the same quota as 100 typical requests. Token-aware rate limiting is the correct model.
What to build
TokenRateLimiter with a sliding window:
-
TokenRateLimiter(max_tokens_per_minute: int, max_requests_per_minute: int) -
check_and_consume(user_id: str, estimated_tokens: int) -> tuple[bool, dict]:(True, {"allowed": True, "tokens_remaining": int, "requests_remaining": int})if within limits(False, {"allowed": False, "reason": str, "retry_after_seconds": float})if exceeded- 60-second sliding window per user
-
get_usage(user_id: str) -> dict
Constraints
- Standard library only. In-memory dicts.
- Sliding window tracks actual timestamps (not fixed-minute buckets).
- Both limits enforced independently.
- Tokens consumed optimistically before the call completes.
Api Async / medium / Step 15 of 23
Async and provider control
Make waiting behavior explicit. Timeouts, retries, and concurrency limits matter more than squeezing everything into one helper.
- - Uses async boundaries coherently
- - Makes timeout and retry decisions legible
- - Would be maintainable under provider instability
- - Is timeout behavior explicit?
- - Is retryable failure separate from terminal failure?
- - Would logs reveal what actually timed out?
Practice
Generate a variation
Generate a new exercise variation to deepen understanding or practice a related concept.
Attempt history
Recent submissions
Before you submit, decide what a strong answer should make obvious to the reviewer.
No attempts yet.