Your personal operating system for career transition.

Build a Document Chunking Pipeline with Overlap and Metadata

Before a document can be embedded and indexed for RAG, it must be split into chunks that fit within the embedding model's context window. Naive splitting on fixed character boundaries produces chunks that cut mid-sentence, lose context at chunk boundaries, and discard metadata needed for filtering. A production chunking pipeline respects sentence boundaries, adds overlap between adjacent chunks, and carries document-level metadata through to each chunk.

What to build

Implement a ChunkingPipeline class with:

1. __init__(chunk_size: int = 500, overlap: int = 50, min_chunk_size: int = 100) — configure chunk size (in characters), overlap (characters shared between adjacent chunks), and minimum chunk size (chunks smaller than this are discarded or merged with the next).

2. chunk_text(text: str) -> list[str] — split text into chunks of approximately chunk_size characters. Prefer to split at sentence boundaries (., !, ? followed by whitespace) rather than mid-word. Each chunk except the first should start with the last overlap characters of the previous chunk.

3. process_document(doc: dict) -> list[dict] — accept a document with id, title, body, and optional metadata: dict. Return a list of chunk dicts, each with:

   • doc_id: the document's id
   • chunk_index: 0-based position
   • text: the chunk text
   • char_start: character offset in the original body
   • metadata: the document's metadata dict (pass through, do not modify)

4. process_batch(docs: list[dict]) -> list[dict] — process multiple documents and return a flat list of all chunks.

Why this matters

Chunk boundary placement directly affects retrieval quality. A chunk that cuts mid-sentence loses context that would have helped the embedding model represent it accurately. Overlap ensures that content near chunk boundaries appears in two adjacent chunks, reducing the chance of a relevant sentence being split across chunks with neither chunk carrying enough context.

Constraints

• If body is missing or empty, return an empty list for that document.
• Sentence boundary detection does not need to be perfect — a simple regex on .!? followed by space is sufficient.
• Chunks shorter than min_chunk_size characters should be dropped (not merged).