AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Build a type-safe provider abstraction layer

Build a Type-Safe Provider Abstraction Layer

Application code that calls OpenAI directly becomes brittle when you add Anthropic as a fallback, switch to a local model for development, or need to mock provider calls in tests. An abstraction layer with a shared interface and typed contracts solves all three.

What to build

Design a provider abstraction that:

Defines a GenerateRequest model — prompt: str, model: str, max_tokens: int, temperature: float, request_id: str.
Defines a GenerateResponse model — request_id: str, text: str, model: str, input_tokens: int, output_tokens: int, latency_ms: int.
Defines an abstract LLMProvider base class — with an abstract async def generate(self, request: GenerateRequest) -> GenerateResponse method and a provider_name: str property.
Implements MockProvider — returns a deterministic fake response based on the prompt hash. Useful for tests.
Implements FallbackProvider — wraps a primary and fallback provider. Tries primary first; on any exception, logs the failure and retries with fallback.
Writes a ProviderRegistry — maps provider names to instances. get(name) -> LLMProvider raises a clear error if the provider is not registered.

Why this matters

This pattern isolates all vendor-specific code behind a stable interface. Tests use MockProvider. Development can use a local model. Production uses a hosted API. The FallbackProvider adds resilience without touching application code.

Constraints

LLMProvider must be an abstract base class (abc.ABC).
FallbackProvider must log a warning when it falls back, including which provider failed and the error.
MockProvider must be deterministic: the same prompt always returns the same text.

Python Ai / medium / Step 4 of 6

Practice stage

General drill

Hint

Keep the solution explicit and reviewable.

Next drill

Implement an LRU cache with TTL for LLM responses

Success criteria

Make the solution explicit, debuggable, and easy to explain.

Review checklist

Review where the boundary is, what gets validated, and what would be hard to debug later.

from __future__ import annotations
import abc
import hashlib
import logging
import time
from pydantic import BaseModel, Field

logger = logging.getLogger(__name__)

class GenerateRequest(BaseModel):
    prompt: str
    model: str = "default"
    max_tokens: int = Field(default=1024, ge=1, le=8192)
    temperature: float = Field(default=0.7, ge=0.0, le=2.0)
    request_id: str = ""

class GenerateResponse(BaseModel):
    request_id: str
    text: str
    model: str
    input_tokens: int
    output_tokens: int
    latency_ms: int

class LLMProvider(abc.ABC):
    @property
    @abc.abstractmethod
    def provider_name(self) -> str: ...

@abc.abstractmethod
    async def generate(self, request: GenerateRequest) -> GenerateResponse: ...

class MockProvider(LLMProvider):
    # TODO: return deterministic fake response based on prompt hash
    @property
    def provider_name(self) -> str:
        return "mock"

async def generate(self, request: GenerateRequest) -> GenerateResponse:
        raise NotImplementedError

class FallbackProvider(LLMProvider):
    # TODO: try primary, fall back to secondary on any exception
    def __init__(self, primary: LLMProvider, fallback: LLMProvider) -> None:
        self._primary = primary
        self._fallback = fallback

@property
    def provider_name(self) -> str:
        return f"fallback({self._primary.provider_name}->{self._fallback.provider_name})"

async def generate(self, request: GenerateRequest) -> GenerateResponse:
        raise NotImplementedError

class ProviderRegistry:
    # TODO: map provider names to instances, raise on missing
    def __init__(self) -> None:
        self._providers: dict[str, LLMProvider] = {}

def register(self, provider: LLMProvider) -> None:
        raise NotImplementedError

def get(self, name: str) -> LLMProvider:
        raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.