AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Add health and readiness probes to an LLM service

Add Health and Readiness Probes to an LLM Service

Kubernetes and cloud load balancers use two distinct probes to manage container lifecycle:

Liveness probe (/health/live): is the container alive? If it fails, kill and restart it.
Readiness probe (/health/ready): is the container ready to serve traffic? If it fails, stop routing requests to it — but do not kill it.

These semantics matter for LLM services. A container can be alive (process running) but not ready (model still loading, provider API unreachable, or token budget exhausted for the day). Conflating liveness and readiness causes cascading restarts and dropped traffic.

What to build

Implement a HealthChecker class that powers both endpoints:

is_alive() -> LivenessResult — checks that the process is running and has not entered a fatal error state. Should almost always return healthy; only fail on unrecoverable conditions.
is_ready() -> ReadinessResult — checks that all dependencies needed to serve requests are available: provider reachability, model loaded, config valid, queue not saturated.
Both methods return a typed result with healthy: bool, checks: dict[str, CheckResult], and latency_ms: int.
Each individual check in is_ready() should time out independently (default 2 seconds) so one slow dependency does not block the whole probe.
The readiness result should report degraded: bool — True when some non-critical checks fail but the service can still partially serve requests.

Constraints

Use only the Python standard library and dataclasses.
Each dependency check is a callable async () -> bool.
Readiness is healthy if all critical checks pass (even if non-critical checks fail).
Implement a check_provider_reachability stub that simulates an async HTTP ping.

Api Async / medium / Step 9 of 23

Practice stage

Async and provider control

Hint

Make waiting behavior explicit. Timeouts, retries, and concurrency limits matter more than squeezing everything into one helper.

Next drill

Enforce per-request token budgets

Success criteria

- Uses async boundaries coherently
- Makes timeout and retry decisions legible
- Would be maintainable under provider instability

Review checklist

- Is timeout behavior explicit?
- Is retryable failure separate from terminal failure?
- Would logs reveal what actually timed out?

from __future__ import annotations
import asyncio
import time
from dataclasses import dataclass, field
from typing import Callable

@dataclass
class CheckResult:
    name: str
    healthy: bool
    latency_ms: int
    error: str = ""

@dataclass
class LivenessResult:
    healthy: bool
    latency_ms: int
    checks: dict[str, CheckResult] = field(default_factory=dict)

@dataclass
class ReadinessResult:
    healthy: bool
    degraded: bool
    latency_ms: int
    checks: dict[str, CheckResult] = field(default_factory=dict)

class HealthChecker:
    def __init__(
        self,
        critical_checks: dict[str, Callable[[], bool]],
        non_critical_checks: dict[str, Callable[[], bool]] | None = None,
        check_timeout_seconds: float = 2.0,
    ):
        # TODO: store checks and timeout
        raise NotImplementedError

async def is_alive(self) -> LivenessResult:
        # TODO: run a minimal process-level check (just confirm process is running)
        raise NotImplementedError

async def is_ready(self) -> ReadinessResult:
        # TODO: run all critical and non-critical checks with timeout
        # healthy if all critical checks pass; degraded if any non-critical fail
        raise NotImplementedError

async def _run_check(self, name: str, check_fn: Callable[[], bool]) -> CheckResult:
        # TODO: run check with timeout, return CheckResult
        raise NotImplementedError

async def check_provider_reachability() -> bool:
    # Stub: simulate a fast async HTTP ping to provider
    await asyncio.sleep(0.01)
    return True

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.