Your personal operating system for career transition.

Build a Data Pipeline with Generator Composition

Document ingestion pipelines need to process large corpora efficiently. Loading an entire dataset into memory before processing creates unnecessary memory pressure and prevents you from streaming results to storage as they are produced. Python generators solve both problems — but only if you compose them correctly.

What to build

Implement a complete document ingestion pipeline using generator composition:

1. read_jsonl(path: str) -> Iterator[dict] — reads a JSONL file line by line, skipping blank lines and logging (not aborting) parse errors.
2. validate_doc(records: Iterator[dict]) -> Iterator[dict] — yields only records that have non-empty "id" and "body" fields (minimum 50 chars). Log skipped records.
3. normalize_doc(records: Iterator[dict]) -> Iterator[dict] — strips whitespace from body, lowercases the "source" field if present, adds "char_count": int.
4. chunk_doc(records: Iterator[dict], max_chars: int = 500) -> Iterator[dict] — splits each document body into chunks of at most max_chars, yielding one chunk dict per chunk with fields: "doc_id", "chunk_index", "text", "metadata".
5. write_jsonl(records: Iterator[dict], path: str) -> int — writes records to a JSONL file and returns the count of records written.
6. run_pipeline(input_path: str, output_path: str, chunk_size: int = 500) -> dict — composes all stages and returns {"docs_read", "docs_valid", "chunks_written"}.

Why this matters

A pipeline that processes one document at a time uses O(1) memory regardless of corpus size. A 10GB JSONL file and a 10MB file should use the same peak memory. Generator composition is the idiomatic Python way to achieve this, and it is used in virtually every production data ingestion pipeline.

Constraints

• Each stage must be a generator function — no materializing lists between stages.
• run_pipeline must work by chaining the generators together, not calling them sequentially.
• The pipeline should process at least one test document end-to-end.