AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Build a health check endpoint for an LLM service

Build a Health Check Endpoint for an LLM Service

A production LLM service needs two health endpoints with different semantics. A naive /ping -> 200 is useless — it does not tell you whether the model is loaded, whether the provider API is reachable, or whether the service can handle a request.

What to build

Implement a HealthChecker class and two endpoint functions:

HealthChecker:
- mark_fatal(reason: str) — marks the service unrecoverable (liveness will fail)
- set_model_loaded(loaded: bool)
- set_provider_healthy(healthy: bool)
- set_cache_healthy(healthy: bool) — non-critical
liveness_check(checker) -> tuple[dict, int]:
- ({"status": "ok"}, 200) unless fatal; ({"status": "fatal", "reason": "..."}, 503) if fatal
readiness_check(checker) -> tuple[dict, int]:
- HTTP 503 if model not loaded OR provider unhealthy
- HTTP 200, degraded=True if only cache is unhealthy
- HTTP 200, degraded=False if all pass

Why this matters

In Kubernetes, liveness failure triggers a container restart; readiness failure only removes the pod from rotation. LLM services loading models at startup need generous startup probes — conflating liveness and readiness causes unnecessary restart loops.

Api Async / medium / Step 14 of 23

Practice stage

Async and provider control

Hint

Make waiting behavior explicit. Timeouts, retries, and concurrency limits matter more than squeezing everything into one helper.

Next drill

Implement token-based rate limiting for an LLM API

Success criteria

- Uses async boundaries coherently
- Makes timeout and retry decisions legible
- Would be maintainable under provider instability

Review checklist

- Is timeout behavior explicit?
- Is retryable failure separate from terminal failure?
- Would logs reveal what actually timed out?

from dataclasses import dataclass, field

@dataclass
class HealthChecker:
    _fatal_reason: str | None = field(default=None, init=False)
    _model_loaded: bool = field(default=False, init=False)
    _provider_healthy: bool = field(default=True, init=False)
    _cache_healthy: bool = field(default=True, init=False)

def mark_fatal(self, reason: str) -> None:
        raise NotImplementedError

def set_model_loaded(self, loaded: bool) -> None:
        raise NotImplementedError

def set_provider_healthy(self, healthy: bool) -> None:
        raise NotImplementedError

def set_cache_healthy(self, healthy: bool) -> None:
        raise NotImplementedError

def liveness_check(checker: HealthChecker) -> tuple[dict, int]:
    # 200 + {"status": "ok"} unless fatal; 503 + {"status": "fatal", "reason": ...}
    raise NotImplementedError

def readiness_check(checker: HealthChecker) -> tuple[dict, int]:
    # 503 if model not loaded OR provider unhealthy
    # 200 + degraded=True if only cache unhealthy
    # 200 + degraded=False if all pass
    raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.