Your personal operating system for career transition.

Build an async model fallback chain with latency budget

Provider reliability is imperfect. A fallback chain means one provider failing does not make your feature fail. A latency budget means a slow provider gets cut off before it degrades user experience.

What you are building

Implement call_with_fallback — an async function that tries models in a priority order and returns the first successful result within the overall latency budget.

Function signature:

async def call_with_fallback(
    request_fn: Callable[[str], Awaitable[dict]],
    models: list[str],
    latency_budget_s: float = 10.0,
    per_call_timeout_s: float = 5.0,
) -> tuple[dict, str]:
    ...
    # Returns: (result_dict, model_name_used)

Behavior:

1. Try each model in models order
2. Each call is wrapped with asyncio.wait_for(..., timeout=per_call_timeout_s)
3. If the overall elapsed time exceeds latency_budget_s, stop and raise TimeoutError
4. On any exception (timeout or provider error), log the failure and try the next model
5. If all models fail, raise RuntimeError("All models in fallback chain exhausted")
6. Return (result, model_name) where model_name is the model that succeeded

Notes

• request_fn takes a model name string and returns a dict with at least {"text": str}
• Track elapsed time with time.monotonic()
• Simulate logging with print(f"[fallback] model={model} error={exc}")