Your personal operating system for career transition.

Parse and validate LLM JSON output with extraction fallback

LLMs return almost-JSON. Your parser needs to handle: raw JSON, markdown-fenced JSON, JSON embedded in prose, and completely invalid output. This exercise builds a robust extractor with Pydantic validation and a fallback path.

What you are building

Implement parse_llm_output(raw_text: str, schema_class: type[BaseModel], fallback: BaseModel) -> BaseModel:

Extraction order:

1. Try json.loads(raw_text.strip()) directly
2. Try extracting from a \``json ... ```code fence (multiline, with optionaljson` tag)
3. Try extracting the first {...} block found with a regex

After extraction:

• If extraction produces a dict, validate with schema_class.model_validate(dict)
• If validation raises ValidationError, return fallback
• If all extraction attempts fail, return fallback

Also implement:

def is_refusal(text: str) -> bool — Returns True if the text appears to be a model refusal. Check for any of these patterns (case-insensitive): "i cannot", "i'm unable", "i don't have", "i can't help", "as an ai".

Example Pydantic schema to test with:

from pydantic import BaseModel

class Sentiment(BaseModel):
    label: str  # "positive", "neutral", "negative"
    score: float  # 0.0 to 1.0
    reason: str