Document Chunking - Splitting Documents for RAG

Status: Active
Last Updated: 2026-08-26
Category: AI/ML - Phase 3: RAG Infrastructure
Prerequisites: embeddings-vector-db, rag-pipeline
Time: 2 hours
Tags: chunking, rag, embeddings, retrieval, preprocessing

Summary

How you cut documents into chunks matters more than which vector database you use. This article covers fixed-size, recursive, and semantic chunking; overlap; metadata attachment; and how to evaluate whether your chunks are actually retrievable.

🎯 What You'll Learn

By the end of this article, you'll be able to:

✅ Explain why naive fixed-length chunking hurts retrieval
✅ Implement recursive character splitting with overlap
✅ Attach metadata that makes filtered retrieval possible
✅ Evaluate chunk quality with a small golden-question set

Context / Why This Matters

The rag-pipeline overview shows retrieval feeding generation; chunking is the step that decides what the retriever can possibly find. Embed a whole page as one vector and every answer dilutes into an average of everything on that page. Embed tiny fragments and no chunk carries enough context to be useful. Chunking is where most homegrown RAG systems silently fail — the pipeline "works", answers are just consistently mediocre.

Chunk output feeds directly into qdrant-setup (storage) and langchain-integration (splitters in practice).

Implementation / Core Content

What Makes a Good Chunk

Self-contained: readable without its siblings. Someone reading only this chunk could tell what it's about.
One topic: a single idea per chunk keeps the embedding focused.
Sized to the answer, not the document: typically 200–500 tokens (~800–2,000 chars) for prose.
Traceable: carries source path, position, and heading metadata.

Strategy 1: Fixed-Size Windows

Split every N characters/tokens regardless of content.

def fixed_chunks(text: str, size: int = 1200, overlap: int = 150) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap
    return chunks

Pros: trivially predictable, uniform embedding cost. Cons: slices sentences and even words in half; a fact spanning a boundary lives fully nowhere. Acceptable as a baseline and for uniformly structured text (logs), otherwise avoid.

Strategy 2: Recursive Character Splitting (Default Choice)

Try natural separators largest-first; fall back progressively:

SEPARATORS = ["\n## ", "\n### ", "\n\n", ". ", " ", ""]

def recursive_chunks(text, max_size=1200, overlap=150):
    for sep in SEPARATORS:
        if sep == "" or sep in text:
            parts = text.split(sep)
            break
    chunks, buf = [], ""
    for part in parts:
        if len(buf) + len(part) + len(sep) <= max_size:
            buf += part + sep
        else:
            if buf:
                chunks.append(buf.strip())
            if overlap and buf:
                tail = buf[-overlap:]
                buf = tail + part + sep      # carry overlap forward
            else:
                buf = part + sep
    if buf.strip():
        chunks.append(buf.strip())
    return chunks

This is conceptually what LangChain's RecursiveCharacterTextSplitter does (langchain-integration). It respects headings first, then paragraphs, then sentences — chunks land on topic boundaries most of the time.

Overlap (10–15% of size) mitigates facts split at boundaries: the same sentence appears at the end of one chunk and start of the next, so either retrieves it. Too much overlap bloats storage and returns near-duplicate hits that crowd out diversity in top-k.

Rules of thumb by document type:

Content	Chunk target	Notes
Markdown / wiki prose	800–1600 chars, split on headings	Prepend heading path to chunk text
Source code	Split on function/class boundaries	Keep whole functions together
Logs	Line-based windows, larger (2–4k)	Timestamps matter; keep chronological runs
Tables/CSV	One row-group per chunk with header repeated	Never split a table mid-row
FAQ / runbooks	One Q&A or procedure per chunk	Natural units already

Structure-Aware Enhancement: Heading Paths

For markdown, prepend the section hierarchy to each chunk's embedded text:

[Runbooks > Backup Failures > Restic exit codes] If restic exits with code 3...

This disambiguates generic sentences ("check the logs") by anchoring them to their section — cheap, high-yield trick.

Strategy 3: Semantic Chunking

Embed individual sentences, then start a new chunk wherever cosine similarity between adjacent sentences drops below a threshold — boundaries follow topic shifts instead of character counts.

import ollama

def semantic_chunks(text, threshold=0.45):
    sents = [s.strip() for s in text.replace("\n", " ").split(". ") if s.strip()]
    vecs = ollama.embed(model="nomic-embed-text", input=sents)["embeddings"]

    def cos(a, b):
        dot = sum(x*y for x, y in zip(a, b))
        return dot / (sum(x*x for x in a) ** .5 * sum(y*y for y in b) ** .5)

    chunks, cur = [], [sents[0]]
    for prev, nxt, v_prev, v_next in zip(sents, sents[1:], vecs, vecs[1:]):
        if cos(v_prev, v_next) < threshold:
            chunks.append(". ".join(cur)); cur = []
        cur.append(nxt)
    chunks.append(". ".join(cur))
    return chunks

Pros: cleanest topical boundaries. Cons: one embedding call per sentence (slow for large corpora), threshold tuning per corpus. Use it when documents are long-form prose with shifting topics; recursive splitting is fine elsewhere.

Metadata Is Half the Job

Every chunk should upsert into Qdrant (qdrant-setup) with:

payload = {
    "source": rel_path,
    "chunk_index": i,
    "heading_path": "Runbooks > Backup Failures",
    "doc_type": "runbook",           # enables filtered retrieval
    "updated": mtime_date_iso,
    "tags": ["backup"],
    "text": chunk_text,
}

Two payload details pay off repeatedly: store chunk_index so neighbors can be fetched (source + chunk_index ± 1) for context expansion after retrieval, and store text itself so search results are self-contained.

Evaluating Chunk Quality

Don't eyeball — measure with a small golden set of ~20 (question → expected source snippet) pairs from real user questions:

def hit_rate(qa_pairs, k=3):
    hits = 0
    for question, expect_substr in qa_pairs:
        results = retrieve(question, k=k)          # embed + qdrant query
        if any(expect_substr in r.payload["text"] for r in results):
            hits += 1
    return hits / len(qa_pairs)

Report hit@k before and after any chunking change:

# baseline recursive 1200/150   -> hit@3: 0.70
# heading-path prepended        -> hit@3: 0.85   <- ship this

Also sanity-check distributions: mean chunk length near target, few chunks under ~50 chars (orphan fragments), no chunk over your max.

Practical Examples

Example 1: Markdown Ingestion With Heading Tracking

import glob, re

def chunk_markdown(path):
    text, base = open(path).read(), []
    chunks, h1, h2 = [], "", ""
    for block in re.split(r"\n(?=#{1,3} )", text):
        m = re.match(r"(#{1,3}) (.+)", block)
        if m:
            level, title = len(m.group(1)), m.group(2)
            if level == 1: h1, h2 = title, ""
            elif level == 2: h2 = title
        body = re.sub(r"^#{1,3} .+\n", "", block).strip()
        if not body:
            continue
        prefix = f"[{h1}" + (f" > {h2}]" if h2 else "]")
        for piece in recursive_chunks(body, max_size=1200, overlap=150):
            chunks.append({"text": f"{prefix} {piece}",
                           "heading_path": f"{h1} > {h2}".strip(" >")})
    return chunks

Each chunk now embeds with its location context attached.

Example 2: Neighbor Expansion at Query Time

Retrieved chunk is relevant but clipped? Pull its siblings:

hit = results[0]
neighbors = client.scroll(
    collection_name="docs",
    scroll_filter=models.Filter(must=[
        models.FieldCondition(key="source", match=models.MatchValue(value=hit.payload["source"])),
        models.FieldCondition(key="chunk_index",
            match=models.MatchRange(gte=hit.payload["chunk_index"] - 1,
                                    lte=hit.payload["chunk_index"] + 1)),
    ]),
)[0]
context = "\n---\n".join(p.payload["text"] for p in sorted(neighbors, key=lambda p: p.payload["chunk_index"]))

This lets you keep chunks small (better precision) without losing surrounding context (recall).

Example 3: Comparing Strategies on Your Own Data

strategies = {
    "fixed":    lambda t: fixed_chunks(t, 1200, 150),
    "recursive": lambda t: recursive_chunks(t, 1200, 150),
    "semantic": lambda t: semantic_chunks(t),
}
for name, fn in strategies.items():
    rebuild_index(fn)                     # clear collection, chunk, embed, upsert
    print(name, f"hit@3={hit_rate(GOLDEN_SET):.2f}")

Expect recursive ≈ semantic > fixed on prose corpora; pick the cheapest winner.

Troubleshooting & Common Pitfalls

Problem	Cause	Fix
Retrieval returns irrelevant half-sentences	Chunks too small / pure fixed splitting	Recursive splitting, ≥800 chars, add heading prefixes
Top-k all near-duplicates	Overlap too large	Reduce overlap to ~10%; dedupe by (source, chunk_index)
Answer needs info from adjacent section	Chunk boundaries split related content	Neighbor expansion; bigger chunks; heading-aware grouping
Whole-doc chunks score mediocre on everything	Vector averaged across topics	Never embed whole pages; chunk first
Tables garbled in answers	Table split mid-row	Detect table blocks, never split; repeat headers
Hit rate fine offline, poor in production	Golden set doesn't match real queries	Harvest real questions continuously; refresh eval set
Re-indexing creates duplicates	Random IDs per run	Deterministic IDs: hash(source + chunk_index)
Semantic chunking too slow	Sentence-level embedding of full corpus	Reserve for long prose; batch embed; cache vectors

Next Steps / Ops Actions

Stand up the store these chunks land in: qdrant-setup
Complete retrieval-to-answer wiring: rag-pipeline
Replace hand-rolled splitters with library equivalents: langchain-integration

Sources & Related

External references consulted:

https://www.pinecone.io/learn/chunking-strategies/
https://qdrant.tech/documentation/advanced-tutorials/ (payload patterns)
https://arxiv.org/abs/2401.10178 (RAG chunking evaluation study)
LangChain text splitters docs (conceptual reference for recursive splitting)

Change Log

2026-08-26

Initial creation by KB build session.