AI Engineer Portal

Your personal operating system for career transition.

Private mode

Exercise

Parse structured LLM output with fallback

Parse Structured LLM Output with Fallback

LLMs are not databases. Even when you ask for JSON, you sometimes get JSON wrapped in markdown fences, JSON with trailing commas, JSON preceded by "Sure! Here is the output:", or valid JSON that does not match the schema you specified. Production-grade parsing must handle all of these cases gracefully.

What you are building

Build a parse_llm_output function that:

Accepts raw model output text and a Pydantic model class
Tries to parse the text as direct JSON
Falls back to extracting JSON from markdown code fences
Falls back to extracting the first JSON object found in the text
Validates the extracted dict against the Pydantic model
If all parsing attempts fail, calls a fallback_factory callable to produce a safe default
Returns a ParseResult with the parsed value, the strategy that worked, and whether it fell back

Why this matters

A structured output parser with explicit fallback behavior is one of the most reused components in an LLM application codebase. Every feature that writes to a database, triggers an action, or renders structured UI needs one. Building it once, correctly, is a force multiplier.

Constraints

Do not use eval() for JSON parsing
Import only the standard library plus pydantic
The fallback_factory should be called with no arguments and return a value that passes Pydantic validation
Log a warning (using the logging module) when a non-primary parse strategy is used

Llm Foundations / medium / Step 2 of 6

Practice stage

General drill

Hint

Keep the solution explicit and reviewable.

Next drill

Implement model routing by task complexity

Success criteria

Make the solution explicit, debuggable, and easy to explain.

Review checklist

Review where the boundary is, what gets validated, and what would be hard to debug later.

from __future__ import annotations

import json
import logging
import re
from dataclasses import dataclass
from typing import Any, Callable, Optional, Type, TypeVar

from pydantic import BaseModel, ValidationError

logger = logging.getLogger(__name__)
T = TypeVar("T", bound=BaseModel)

@dataclass
class ParseResult:
    value: Any
    strategy: str  # "direct", "fence", "extracted", "fallback"
    used_fallback: bool

def parse_llm_output(
    raw_text: str,
    model_cls: Type[T],
    fallback_factory: Callable[[], T],
) -> ParseResult:
    """
    Parse raw LLM output into a validated Pydantic model with graceful fallback.

Strategies tried in order:
    1. Direct JSON parse of the full text
    2. Extract JSON from markdown code fences (```json ... ```)
    3. Extract the first JSON object found with regex
    4. Call fallback_factory() and return with used_fallback=True
    """
    # TODO: implement strategy 1 - direct JSON parse
    # TODO: implement strategy 2 - code fence extraction
    # TODO: implement strategy 3 - regex JSON object extraction
    # TODO: implement fallback strategy
    # Hint: each strategy should try json.loads() then model_cls(**data)
    # and catch both json.JSONDecodeError and ValidationError
    raise NotImplementedError

def _try_parse(raw: str, model_cls: Type[T]) -> Optional[T]:
    """Try to parse raw string as JSON and validate against model_cls. Return None on failure."""
    # TODO: implement
    raise NotImplementedError

Practice

Generate a variation

Generate a new exercise variation to deepen understanding or practice a related concept.

Attempt history

Recent submissions

Before you submit, decide what a strong answer should make obvious to the reviewer.

No attempts yet.