Why this matters
Most LLM application tutorials assume text-in, text-out. Production AI applications are rarely that clean. Users upload PDFs. Support flows include screenshots. Voice interfaces need speech. Documents contain tables that break if you treat them as prose. Multi-modal inputs are the norm, not the exception, and engineers who can build robust pipelines for them have a significant advantage.
This lesson covers the patterns you need for the most common multi-modal scenarios: vision APIs, audio transcription and synthesis, document understanding, and chaining these together into a coherent pipeline.
Core concepts
Vision APIs: sending images to models
Claude and GPT-4V accept images as part of the messages array. Images are either passed as base64-encoded data or as publicly accessible URLs. Base64 is more reliable in production (no latency from fetching remote URLs, no dependency on image availability).
import anthropic
import base64
from pathlib import Path
def describe_image(image_path: str, question: str = "What is in this image?") -> str:
"""Send an image to Claude and get a description."""
client = anthropic.Anthropic()
# Load and encode the image
image_data = Path(image_path).read_bytes()
b64_image = base64.standard_b64encode(image_data).decode("utf-8")
# Detect media type
suffix = Path(image_path).suffix.lower()
media_type_map = {
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".png": "image/png",
".gif": "image/gif",
".webp": "image/webp",
}
media_type = media_type_map.get(suffix, "image/png")
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=1024,
messages=[
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": media_type,
"data": b64_image,
},
},
{
"type": "text",
"text": question,
},
],
}
],
)
return response.content[0].text
# Document understanding: extract structured data from an image
def extract_invoice_data(image_path: str) -> dict:
"""Extract structured invoice fields from an image using vision + structured output."""
client = anthropic.Anthropic()
image_data = Path(image_path).read_bytes()
b64_image = base64.standard_b64encode(image_data).decode("utf-8")
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=1024,
tools=[
{
"name": "extract_invoice",
"description": "Extract structured data from an invoice image",
"input_schema": {
"type": "object",
"properties": {
"vendor_name": {"type": "string"},
"invoice_number": {"type": "string"},
"invoice_date": {"type": "string", "description": "ISO 8601 date"},
"total_amount": {"type": "number"},
"currency": {"type": "string"},
"line_items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"description": {"type": "string"},
"quantity": {"type": "number"},
"unit_price": {"type": "number"},
"total": {"type": "number"},
}
}
}
},
"required": ["vendor_name", "invoice_number", "total_amount"]
}
}
],
tool_choice={"type": "tool", "name": "extract_invoice"},
messages=[
{
"role": "user",
"content": [
{
"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": b64_image},
},
{
"type": "text",
"text": "Extract all invoice data from this image."
}
]
}
]
)
tool_use = next(b for b in response.content if b.type == "tool_use")
return tool_use.input
Audio: speech-to-text and TTS integration
The OpenAI Whisper API handles transcription with high accuracy across languages. The TTS API converts text back to audio. In production, you typically use transcription at the ingestion boundary (convert audio to text, then feed the text pipeline) and TTS at the delivery boundary (convert the final text response to audio for playback).
import openai
from pathlib import Path
def transcribe_audio(audio_path: str, language: str = "en") -> dict:
"""Transcribe audio to text using Whisper.
Returns both the full transcript and segment-level timestamps,
which are useful for syncing audio with transcript display.
"""
client = openai.OpenAI()
with open(audio_path, "rb") as audio_file:
response = client.audio.transcriptions.create(
model="whisper-1",
file=audio_file,
language=language,
response_format="verbose_json", # includes segments + timestamps
timestamp_granularities=["segment"]
)
return {
"text": response.text,
"language": response.language,
"duration_seconds": response.duration,
"segments": [
{
"start": seg.start,
"end": seg.end,
"text": seg.text.strip(),
}
for seg in (response.segments or [])
],
}
def text_to_speech(text: str, output_path: str, voice: str = "alloy") -> None:
"""Convert text to speech and save to file.
Voices: alloy, echo, fable, onyx, nova, shimmer
"""
client = openai.OpenAI()
response = client.audio.speech.create(
model="tts-1",
voice=voice,
input=text,
)
Path(output_path).write_bytes(response.content)
Key integration patterns:
- For real-time voice interfaces, stream the TTS response to avoid buffering the full audio before playback begins
- Whisper handles background noise well but struggles with overlapping speakers — use diarization (speaker separation) as a pre-processing step for meeting recordings
- Cache TTS outputs for static content (FAQ answers, onboarding narration) to avoid repeated API calls
Document processing: PDF extraction and OCR
PDFs are visual layout files, not text files. A PDF extractor must reconstruct reading order, handle multi-column layouts, and preserve table structure — none of which is guaranteed. For AI pipelines, the goal is clean text with metadata (page number, section heading, table data preserved).
import pdfplumber
from pathlib import Path
from dataclasses import dataclass, field
@dataclass
class DocumentPage:
page_number: int
text: str
tables: list[list[list[str]]] = field(default_factory=list)
image_count: int = 0
def extract_pdf(pdf_path: str) -> list[DocumentPage]:
"""Extract text and tables from a PDF, preserving structure."""
pages = []
with pdfplumber.open(pdf_path) as pdf:
for i, page in enumerate(pdf.pages):
# Extract text with layout preservation
text = page.extract_text(x_tolerance=3, y_tolerance=3) or ""
# Extract tables as structured data
tables = []
for table in page.extract_tables():
# Filter empty rows, clean cell whitespace
clean_table = [
[cell.strip() if cell else "" for cell in row]
for row in table
if any(cell for cell in row)
]
if clean_table:
tables.append(clean_table)
pages.append(DocumentPage(
page_number=i + 1,
text=text,
tables=tables,
image_count=len(page.images),
))
return pages
def table_to_markdown(table: list[list[str]]) -> str:
"""Convert a table (list of rows) to a markdown table string."""
if not table:
return ""
header = table[0]
separator = ["---"] * len(header)
rows = table[1:]
lines = [
"| " + " | ".join(header) + " |",
"| " + " | ".join(separator) + " |",
]
for row in rows:
lines.append("| " + " | ".join(row) + " |")
return "\n".join(lines)
For scanned documents (images of text, not machine-readable PDFs), you need OCR. pytesseract is the standard open-source option; for production accuracy, cloud OCR (Google Document AI, AWS Textract, Azure Form Recognizer) is significantly better.
Building a multi-modal pipeline
The common pattern: image input → structured extraction → action or further processing.
import anthropic
import base64
from pathlib import Path
from pydantic import BaseModel
from typing import Optional
class ExtractedFormData(BaseModel):
form_type: str
fields: dict[str, str]
confidence: float # 0.0 - 1.0
needs_human_review: bool
review_reason: Optional[str] = None
def process_form_image(image_path: str) -> ExtractedFormData:
"""
Multi-modal pipeline: image → structured extraction → validation → routing.
This pipeline:
1. Sends the image to Claude with a structured extraction tool
2. Validates the extracted data
3. Flags low-confidence extractions for human review
"""
client = anthropic.Anthropic()
image_bytes = Path(image_path).read_bytes()
b64 = base64.standard_b64encode(image_bytes).decode()
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=1024,
tools=[
{
"name": "extract_form",
"description": "Extract all fields from a form image",
"input_schema": {
"type": "object",
"properties": {
"form_type": {
"type": "string",
"description": "Type of form: invoice, receipt, application, contract, other"
},
"fields": {
"type": "object",
"description": "Key-value pairs of all extractable fields",
"additionalProperties": {"type": "string"}
},
"confidence": {
"type": "number",
"description": "Confidence 0.0-1.0 in extraction accuracy"
},
"needs_human_review": {
"type": "boolean",
"description": "True if handwriting, damage, or ambiguity reduces reliability"
},
"review_reason": {
"type": "string",
"description": "Explain why human review is needed, if applicable"
}
},
"required": ["form_type", "fields", "confidence", "needs_human_review"]
}
}
],
tool_choice={"type": "tool", "name": "extract_form"},
messages=[
{
"role": "user",
"content": [
{
"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": b64},
},
{
"type": "text",
"text": (
"Extract all fields from this form. "
"Be conservative with confidence — flag anything unclear for review."
),
},
],
}
],
)
tool_use = next(b for b in response.content if b.type == "tool_use")
return ExtractedFormData.model_validate(tool_use.input)
def route_extracted_form(data: ExtractedFormData) -> str:
"""Route the extracted form based on type and confidence."""
if data.needs_human_review or data.confidence < 0.85:
return f"human_review_queue: {data.review_reason or 'low confidence'}"
if data.form_type == "invoice":
return "accounts_payable_pipeline"
if data.form_type == "receipt":
return "expense_report_pipeline"
return "general_documents_queue"
Common mistakes
Sending full-resolution images for simple tasks. Vision API costs scale with image size (token count). For most document extraction tasks, a 1024x1024 image is sufficient. Resize before sending — most providers accept base64 images and will resize internally, but you pay for the full transfer.
Not handling image decoding failures gracefully. Image bytes from user uploads are often corrupt, truncated, or in unsupported formats. Always catch exceptions when loading images and return a structured error rather than crashing the pipeline.
Treating audio transcription as infallible. Whisper is excellent but makes errors on domain-specific terminology, proper nouns, and strong accents. For high-stakes transcription (legal, medical), build a post-processing step that flags low-confidence words (available in the verbose_json response format).
Mixing table data into prose chunks for RAG. If you extract a PDF and feed the whole page as a single text chunk, table rows become meaningless strings like "Q1 Q2 Q3 Q4 1200 1450 1380 1620". Extract tables separately, convert to structured JSON or markdown, and index them with metadata indicating they are tabular data.
Skipping multi-modal eval. Vision extraction pipelines are harder to eval than text pipelines because ground truth requires human annotation of the image. Build a small golden set of annotated images (20–50 examples) before deploying a vision extraction feature.
Try it yourself
Build a document understanding pipeline that takes a PNG screenshot of a simple data table (you can create one in a spreadsheet and screenshot it) and extracts the data as a Python list of dicts. The pipeline should:
- Accept a file path to the image
- Send it to a vision model with a structured extraction tool
- Return a
list[dict]where each dict is one row of the table, using the header row as keys - Handle the case where the model is uncertain about a cell value by using
Noneinstead of guessing
Test it on at least two different table screenshots and verify the extracted data matches the source.