Chunk your documents for RAG

Chunking is splitting a document into passages small enough to retrieve one at a time and large enough to mean something on their own. Start with structure-aware splitting at around 400 to 800 tokensA token is a chunk of text the model counts in, roughly three quarters of a word in English. 800 tokens is about 600 words. per chunk, add 10 to 20 percent overlap, and attach the source path and section heading to every chunk. The rest of this guide explains why each of those numbers is the default and when to move off it.

If you have not built the surrounding pipeline yet, read RAG fundamentals first. Chunking is the first stage, and it sets the ceiling for everything after it.

Why does chunk size decide retrieval quality?

Retrieval works by comparing the meaning of a question to the meaning of each stored passage. The chunk is the unit you store, search, and hand back to the model. Get the unit wrong and no embedding model, vector store, or reranker can recover.

Two failure shapes bracket the problem:

• Chunks too large. A 4,000-token chunk covers four or five topics. Its single meaning vector is the average of all of them, so it matches every question weakly and no question strongly. When it does get retrieved, it floods the model's context with text the question never asked about.
• Chunks too small. A one-sentence chunk loses the context that makes it answerable. "It shipped on Tuesday" is useless without knowing what "it" is. Tiny chunks also multiply your index size and retrieval count for no gain.

The job is to make each chunk a self-contained idea: big enough to stand alone, small enough to be about one thing.

Pick a size, then a strategy

Begin with a target size, then choose how you cut to hit it.

What chunk size should I start with?

For mixed prose like documentation, support articles, and internal wikis, a target of 400 to 800 tokens per chunk is the working default. That is roughly one to three paragraphs, which is usually one coherent point. Adjust by document type:

Measure in tokens, not characters, so your numbers line up with the model's context budget and the embedding model's input limit.Common gotchaA common slip is to size chunks by character or word count, then quietly blow past the embedding model's token limit.The model truncates the overflow, so the tail of every long chunk never gets embedded and silently disappears from search. Count with the same tokenizer your embedding model uses.

Which chunking strategy should I use?

Three strategies cover the common cases. Use the simplest one that holds up on your data.

• Fixed-size. Cut every N tokens. Simple and predictable, but blind to structure, so it slices mid-sentence and mid-table. Acceptable only for flat, uniform text.
• Recursive splitting. Split on the largest natural boundary first (sections), then paragraphs, then sentences, until each piece is under the size limit. This respects how the document is written and is the default for prose. It balances quality against cost.
• Semantic chunking. Embed sentences and cut where the topic shifts. Higher quality on uneven documents, but it costs more to build because every sentence gets embedded up front. Reserve it for research papers, contracts, and dense technical text where a topic shift mid-paragraph is common.

Recursive splitting is where to start. Move to semantic chunking only when you can measure that recursive boundaries are hurting retrieval on your corpus.

Overlap keeps boundary facts alive

A fact that lands on a chunk boundary can be cut in half: the setup ends one chunk and the payoff starts the next, so neither chunk answers the question. Overlap fixes this by repeating the tail of one chunk at the head of the next.

Set overlap to 10 to 20 percent of chunk size. For an 800-token chunk that is 80 to 160 tokens of shared text. Enough that a sentence spanning the cut survives intact in at least one chunk, not so much that you bloat the index with duplicates.

Increase overlap when concepts routinely span boundaries, such as numbered steps that reference the step before. Decrease it when chunks are already self-contained, such as FAQ answers, where overlap only adds noise.Common gotchaThe tempting move is to crank overlap to 50 percent or more so nothing ever falls on a boundary. It backfires: the top results come back as near-duplicate neighbours that crowd out different passages, and your index swells with repeated text you pay to store and embed.Keep overlap at 10 to 20 percent and fix true boundary misses by cutting on a larger structural unit instead.

Respect the document's own structure

The best boundary is the one the author already wrote. Cutting on a heading, a list, or a code block keeps a complete idea together far better than cutting at an arbitrary token count.

• Markdown and HTML. Split on heading levels first. A section under an h2 is usually one topic. Keep tables and code fences whole, since splitting them destroys their meaning.
• Code. Split on function and class boundaries, not line counts, so a retrieved chunk is a unit a reader can use. See RAG for coding agents for how code retrieval differs from prose.
• PDFs and slides. Page and slide boundaries are weak signals. Extract the text, recover the heading hierarchy where you can, then split recursively.

When a single section blows past your size limit, fall back to paragraph then sentence splits inside it. That is exactly what recursive splitting does for you.

What metadata should I attach to each chunk?

A chunk without context is hard to retrieve and impossible to cite. Store structured fields alongside the text so you can filter searches and show the model where each passage came from.

Metadata is also how you scope a search. Filtering to one product area or one date range before the vector search narrows the field to passages that can plausibly answer, which lifts precision more than tuning chunk size.

Make small chunks carry their context

A short chunk loses the document it came from. Two techniques put that context back.

The simpler one is to prepend the document title and section heading to the chunk text before you embed it. "Refund window. Customers have 30 days." embeds far closer to a refund question than "Customers have 30 days." alone.

The stronger one is contextual retrieval: before embedding, generate a short explanatory blurb (50 to 100 tokens) that situates the chunk within the whole document, and prepend that. Anthropic reports this cuts retrieval failures by 35 percent on its own, by 49 percent combined with keyword (BM25A classic keyword-ranking algorithm that scores documents by exact word matches and how rare those words are. Strong on exact terms, blind to meaning.) search, and by up to 67 percent once a reranker is added on top. You generate those blurbs once, at index time, with Claude.

This is a job to hand to the Claude Code CLI. Point it at your corpus and let it write the indexing script, with the contextual blurb step built in:

Let Claude write and run the pipeline code. You stay in charge of the strategy: the size, the overlap, the boundaries, and what counts as good retrieval.

How do I test my chunking?

Chunking is a tuning problem, so measure it instead of guessing. Build a small set of real questions with the chunk you expect each one to retrieve, then check whether your settings return it.

Read the misses, not only the score. The pattern tells you what to change:

• Answers cut in half across two chunks means raise overlap or cut on a larger boundary.
• The right topic buried in a chunk full of other topics means shrink the target size.
• Tiny fragments that match nothing useful mean raise the size or add the title-and-heading prefix.

Re-index after each change and rerun the same evaluation. One variable at a time, so you can attribute every move.

A starting recipe

When you have no signal yet, ship these defaults and let the evaluation move you off them:

• Recursive splitting on headings, then paragraphs, then sentences.
• 700-token target, 15 percent overlap.
• Tables and code fences kept whole.
• source, heading, and position metadata on every chunk.
• Title and section heading prepended to the chunk text before embedding.
• A 30-question evaluation set checked on recall@5.

That recipe retrieves well on documentation out of the box. Add contextual blurbs and a reranker when the evaluation says the easy wins are gone.

Once chunking is solid, the next lever is how you search the index. Hybrid search and reranking build on the foundation laid in RAG fundamentals; good chunks make both of them work harder for you.