Chunking strategy
Chunking defines the unit your retrieval system can reason about. Good chunking preserves semantic coherence and provenance.
Poor chunking creates downstream problems that look like:
- irrelevant retrieval
- weak citations
- duplicated answers
- missing context
What chunking is really doing
You are deciding:
- how much context belongs together
- what metadata travels with it
- what your retriever can later rank, filter, and cite
So chunking is not a preprocessing footnote. It is a product decision.
What makes a good chunk
A good chunk:
- preserves one coherent idea
- carries useful provenance
- is large enough to answer a question
- is small enough to rank precisely
Boundary choices that matter
Prefer natural boundaries when possible:
- headings
- section blocks
- paragraphs that stay on one topic
- structured records with meaningful fields
Avoid arbitrary slicing that breaks concepts mid-thought unless you also have a strong overlap strategy.
Metadata is part of chunk design
A chunk without useful metadata is harder to trust later.
Useful metadata often includes:
- source identifier
- title or section name
- document date
- content type
- tags that matter for filtering
Decision guide
Use smaller chunks when:
- precision matters more than breadth
- users ask targeted factual questions
- citation quality matters a lot
Use larger chunks when:
- context is highly interdependent
- users need broader synthesis
- the document structure would break if sliced too aggressively
What to inspect in practice
If retrieval feels weak, inspect:
- top chunks for one real query
- whether the chunk boundaries preserved meaning
- whether adjacent chunks should have stayed together
- whether metadata could have filtered better candidates upward
Common anti-patterns
- chunking only by character count
- forgetting metadata entirely
- keeping chunks so large that ranking becomes fuzzy
- keeping chunks so small that the model loses context
Practical takeaway
Chunking should be reviewed the same way you review an API contract: does it preserve meaning, and does it support the later behaviors you care about?