Document ingestion and chunking
Why this matters
The most common reason a RAG system gives bad answers has nothing to do with the LLM and nothing to do with your vector database. It is because the chunks entering the retrieval index are either too big, too small, semantically broken in the middle of a sentence, or stripped of the metadata that would let the retrieval layer make good decisions.
Chunking is the first place where the entire system can go wrong silently. The vector DB will happily index garbage chunks and return them with high cosine similarity scores. The model will faithfully summarize whatever text it receives. Nobody throws an exception. You just get bad answers and no signal about why.
As a full-stack engineer, think of chunking like normalization in relational databases: the decisions you make here constrain everything downstream. Get them right early and retrieval becomes tractable. Get them wrong and you will spend weeks chasing phantom quality problems.
Core concepts
Fixed-size chunking. Split text every N characters or tokens with an optional overlap. Fast, simple, predictable. Works well for structured documents where content density is uniform — think product catalogs, legal clauses, or FAQ entries where each item is roughly the same size.
The catch: fixed splits cut sentences in half. A chunk boundary mid-sentence means neither chunk contains a complete thought, and retrieval may return the second half without the premise.
Recursive character chunking. Split on natural boundaries first (paragraphs, then sentences, then words) until chunks reach the target size. This is what LangChain's RecursiveCharacterTextSplitter implements. Better than fixed-size for most prose text because it respects sentence and paragraph structure.
Semantic chunking. Use embedding similarity between consecutive sentences to detect topic shifts, then split at boundaries where similarity drops. Expensive to compute but produces chunks that each contain a single coherent idea — which is exactly what retrieval needs. Worth the cost for high-stakes content like medical or legal documents.
Chunk overlap. The overlap parameter copies the last N tokens of one chunk into the start of the next. This prevents key context from disappearing when a relevant passage spans a chunk boundary. A 15–20% overlap is a common starting point. Too much overlap bloats your index; too little causes boundary blindness.
Metadata extraction. Every chunk should carry metadata that answers: where did this come from? (source, page, section), when was it created? (timestamp, version), what type is it? (heading, body, code, table). This metadata is not used for semantic search — it is used for filtering, citations, and debugging. Without it, you cannot answer "show me the source" or "only search docs from this year."
Handling different file formats. PDFs are the worst. A PDF is a visual layout format, not a text format. Extraction tools often produce column-order confusion, header/footer noise, and broken hyphenation. Markdown and HTML are better because structure is explicit. For code, respect function and class boundaries rather than splitting mid-function. For tables, keep the header row in every chunk that contains table data.
Working example
from __future__ import annotations
import re
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class Chunk:
text: str
metadata: dict = field(default_factory=dict)
def fixed_chunker(
text: str,
chunk_size: int = 512,
overlap: int = 64,
source: str = "",
) -> list[Chunk]:
"""
Simple fixed-size chunker with overlap.
chunk_size and overlap are measured in characters.
"""
chunks = []
start = 0
doc_index = 0
while start < len(text):
end = min(start + chunk_size, len(text))
chunk_text = text[start:end].strip()
if chunk_text:
chunks.append(Chunk(
text=chunk_text,
metadata={
"source": source,
"chunk_index": doc_index,
"char_start": start,
"char_end": end,
},
))
doc_index += 1
start += chunk_size - overlap # step forward by (size - overlap)
return chunks
def recursive_chunker(
text: str,
max_size: int = 512,
overlap: int = 64,
source: str = "",
) -> list[Chunk]:
"""
Recursive chunker: splits on paragraphs, then sentences, then words.
Prefers natural boundaries over arbitrary character counts.
"""
separators = ["\n\n", "\n", ". ", " ", ""]
def split(text: str, sep_index: int) -> list[str]:
if len(text) <= max_size or sep_index >= len(separators):
return [text]
sep = separators[sep_index]
parts = text.split(sep) if sep else list(text)
result, current = [], ""
for part in parts:
candidate = (current + sep + part).strip() if current else part
if len(candidate) <= max_size:
current = candidate
else:
if current:
result.append(current)
# If this single part is still too big, recurse
if len(part) > max_size:
result.extend(split(part, sep_index + 1))
current = ""
else:
current = part
if current:
result.append(current)
return result
raw_chunks = split(text, 0)
chunks = []
for i, chunk_text in enumerate(raw_chunks):
if chunk_text.strip():
chunks.append(Chunk(
text=chunk_text.strip(),
metadata={"source": source, "chunk_index": i},
))
return chunks
# --- Demonstrate on a sample document ---
sample = """
Introduction to Vector Databases
Vector databases store high-dimensional embeddings and enable semantic search. Unlike traditional databases, they find the nearest neighbors by distance, not exact match.
Why this matters for AI applications is straightforward: language models convert text into dense vectors. To retrieve the most relevant context, you need a store that can find the closest vectors efficiently.
Common vector databases include Pinecone, Chroma, Weaviate, and pgvector. Each makes different tradeoffs between query speed, accuracy, and operational complexity.
"""
chunks = recursive_chunker(sample, max_size=200, source="intro.md")
for c in chunks:
print(f"[{c.metadata['chunk_index']}] ({len(c.text)} chars) {c.text[:80]}...")
Running this produces chunks aligned to paragraph boundaries rather than arbitrary character positions — so each chunk contains a complete thought.
Common mistakes
-
Ignoring chunk size distribution. A healthy corpus has a tight distribution of chunk sizes. If you have some chunks that are 50 tokens and others that are 2000 tokens, retrieval will systematically prefer longer chunks (they contain more terms) even when shorter chunks are more relevant. Visualize your chunk size histogram before indexing.
-
No overlap for dense technical content. For API documentation or legal text where every sentence matters, zero overlap means context at chunk boundaries is lost. Start with 10–20% overlap.
-
Stripping all metadata. Engineers often write a fast ingestion script that extracts the text and ignores headers, page numbers, section titles, and timestamps. Then they realize they cannot answer "what page is this from" or filter by date range.
-
Chunking PDFs without cleaning first. Raw PDF extraction often includes headers, footers, page numbers, and navigation text in the middle of content. Clean before chunking, not after.
-
One chunk size for all document types. A FAQ answer and a technical specification are different shapes of text. Use smaller chunks (256–512 tokens) for factual lookups and larger chunks (512–1024 tokens) for explanatory content where context matters more.
Try it yourself
Take a document you use at work — a README, an internal wiki page, or a PDF spec sheet. Run both the fixed-size and recursive chunkers on it. Print each chunk with its character count. Count how many chunks are cut mid-sentence in the fixed-size version. Now add a metadata field for the section heading by detecting lines that end with a colon or match a heading pattern. Can you filter chunks by section heading in a retrieval step?