AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Implement an LRU cache with TTL for LLM responses

Implement an LRU Cache with TTL for LLM Responses

LLM calls are expensive and slow. For many AI features — documentation lookups, FAQ answering, static content generation — the same prompt produces the same answer. Caching identical prompts eliminates redundant API calls and dramatically reduces latency for repeated queries.

What to build

Implement a LLMResponseCache class that combines LRU eviction with TTL expiry:

LRU eviction — when the cache reaches max_size, evict the least-recently-used entry (not the oldest by insertion time).
TTL expiry — each entry expires after ttl_seconds. Expired entries are treated as cache misses and removed on access.
Deterministic cache key — cache_key(prompt: str, model: str, temperature: float) -> str returns a stable SHA-256 hash. Same inputs must always produce the same key.
get(key: str) -> dict | None — returns the cached value if present and not expired, else None. Accessing an entry updates its LRU position.
set(key: str, value: dict) -> None — stores the value. If at capacity, evict the LRU entry first.
stats() -> dict — returns {"size": int, "hits": int, "misses": int, "evictions": int, "hit_rate": float}.

Why this matters

Caching is the single highest-leverage optimization in most AI applications. Understanding how LRU + TTL interact is essential for building caches that stay fresh without consuming unbounded memory.

Constraints

Use collections.OrderedDict to implement LRU ordering — do not use functools.lru_cache (that hides the mechanism).
Do not import any third-party caching library.
hit_rate should be hits / (hits + misses), returning 0.0 when no requests have been made.

Python Ai / medium / Step 5 of 6

Practice stage

General drill

Hint

Keep the solution explicit and reviewable.

Next drill

Profile and optimize a token-heavy pipeline

Success criteria

Make the solution explicit, debuggable, and easy to explain.

Review checklist

Review where the boundary is, what gets validated, and what would be hard to debug later.

from __future__ import annotations
import hashlib
import json
import time
from collections import OrderedDict
from typing import Optional

class LLMResponseCache:
    def __init__(self, max_size: int = 500, ttl_seconds: float = 3600) -> None:
        self._max_size = max_size
        self._ttl = ttl_seconds
        self._store: OrderedDict[str, tuple[dict, float]] = OrderedDict()
        self._hits = 0
        self._misses = 0
        self._evictions = 0

@staticmethod
    def cache_key(prompt: str, model: str, temperature: float) -> str:
        # TODO: return stable SHA-256 hash of (prompt, model, temperature)
        raise NotImplementedError

def get(self, key: str) -> Optional[dict]:
        # TODO: return value if present and not expired, update LRU order
        raise NotImplementedError

def set(self, key: str, value: dict) -> None:
        # TODO: store value, evict LRU entry if at capacity
        raise NotImplementedError

def stats(self) -> dict:
        # TODO: return size, hits, misses, evictions, hit_rate
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.