How do you split documents for RAG?

Chunking means cutting each document into pieces of the size retrieval will hand back. Start at 200 to 500 tokens, with a little overlap when the cut is blind. But the number is the least important part: what sets the quality is cutting along the structure of the text — section, clause, table row — instead of counting characters until the document runs out.

The chunk is the most consequential piece of RAG architecture because it carries two jobs that pull in opposite directions. It is the unit of retrieval, and for that it wants to be small: a vector standing in for five thousand tokens stands in for nothing in particular. It is also the text the model reads to answer, and for that it wants to be large enough to contain the whole answer.

That tension is the entire subject. Nearly every chunking technique that has appeared since 2023 — contextual retrieval, late chunking — exists to get the recall of a small chunk without paying the loss of context that comes with it. And with PDFs the problem starts earlier: you are not cutting the document, you are cutting whatever the parser managed to pull out of it.

The four strategies

Fixed size counts characters or tokens and cuts. It is the most criticised approach and the one every project should use first, for a pragmatic reason: it is deterministic and cheap, so it gives you a baseline. Without a baseline you cannot tell whether the clever strategy improved anything.

By paragraph cuts on the blank line. It works well on prose and badly on anything with structure: a technical document has two-line paragraphs and forty-line tables, and cutting on blank lines produces chunks of wildly different sizes.

By section uses the heading hierarchy. It has the best effort-to-result ratio across most corporate corpora, because that hierarchy was already designed by somebody who knew what belonged together. A 2024 study that chunked financial reports by structural element type rather than by paragraph reported better RAG results and — the detail that matters — a chunk size that comes out good without tuning¹.

Semantic embeds sentence by sentence and cuts where the distance between neighbouring sentences jumps, on the assumption that the jump marks a change of topic. It is the most marketed strategy and the one with the thinnest evidence. A 2024 study evaluated semantic chunking on document retrieval, evidence retrieval and retrieval-based answer generation, and concluded that its computational cost is not justified by consistent gains over fixed-size splitting².

What actually runs in most pipelines is a blend of these: the recursive split. You define a list of separators in order of strength — section break, paragraph break, line break, sentence end, space — and try the strongest first. If the resulting piece is still over the limit, drop a level and try again inside it. That is why a decent splitter is thirty lines rather than one: cutting by character is the last resort, what is left when no better separator turned up.

There is a fifth approach, and it costs more: let an LLM decide where to cut. LumberChunker, from 2024, walks groups of passages asking the model where the content starts to shift, and reported a 7.37% retrieval gain (DCG@20) over the strongest baseline on a benchmark of a hundred Project Gutenberg books³. The scope matters: long-form narrative, where topic boundaries are genuinely hard to detect by rule. In documentation with headings, you already have the boundary for free.

What the size actually controls

Small chunks retrieve better. That is not intuition: a 2023 study compared indexing granularities and showed that indexing by units finer than a passage — propositions, defined as self-contained atomic expressions — beats passage-level units at retrieval, and improves downstream question answering under a fixed compute budget⁴.

The reason is mechanical. An embedding is a compressed average of the whole chunk’s text. The more topics you pack in, the more the vector becomes the centre of mass of everything and the less it sits near any specific question. A chunk holding the refund policy, the delivery window and the support hours is not close to any of those three questions.

And small chunks answer worse, for the opposite reason. The passage arrives at the model stripped of what surrounded it. “The window is 30 days” is true, retrievable and useless without the previous sentence, which said which window.

So size is not a quality choice. It is where you place that trade-off. And the question that resolves it is not “what is the best size” but “how much contiguous text does a typical answer in my corpus need”. In an FAQ, one sentence. In a contract, the clause plus its exceptions. In code, the whole function with its signature.

There is a way out of choosing at all, and it is simpler than it sounds: index at one size and serve at another. You build the vector from the small chunk, which retrieves well, and at prompt-assembly time you expand the retrieved passage to include the paragraph before and after it, or the whole section it came from. The price is storing each chunk’s offset inside its document — two integers per chunk — and it dismantles most of the ideal-size argument.

The damage from a blind cut

Tables are the easiest case to see, which is why they are worth spelling out. A pricing table keeps the meaning of its cells in the header. Cut through the middle and the second half becomes a list of values with no column names. The chunk is still retrievable — “Pro” and “50 GB” are right there, a question about the Pro plan matches it — and the model receives “Pro | 50 GB | $2/GB” with no idea what each field is.

The same thing happens less visibly in prose. A clause severed from the paragraph saying “except in the cases listed in 4.2”. A list item without the heading that said what the list was. A code block without the line that said which file it belongs to.

None of these raise an error. All of them produce a well-written, partly wrong answer, which is the most expensive way to be wrong, because nobody reviews what looks right.

Overlap: what it buys and what it costs

Overlap means repeating the end of one chunk at the start of the next. It buys one specific thing: it guarantees that a sentence landing exactly on a boundary survives whole in at least one of the two chunks. Ten to twenty per cent of the chunk size covers most cases.

The cost shows up in three places. You index more vectors, in proportion to the overlap. You pay embedding on the repeated text. And neighbouring chunks become more similar to each other, which raises the odds of your top-5 arriving with two near-identical chunks occupying two of the five slots.

The practical conclusion is dull and true: overlap is a patch for a blind cut. When the cut follows sections or clauses, the boundary already sits somewhere nobody ends a sentence halfway, and overlap becomes waste with an extra step.

Where it breaks

The embedding model’s token limit. Every model has a maximum input length. Text beyond it gets truncated, and most APIs truncate silently. A 2,000-token chunk fed to a 512-token model becomes a vector of the first 512 tokens, and you end up with an index where the tail of every chunk simply does not exist. It is the quietest failure on this list.

Large chunks push the cost onto every question. Doubling the chunk size doubles the context on every query, forever. Indexing happens once; the prompt happens every time.

Boilerplate contaminates every vector. A confidentiality footer, a header with the company name, a three-line legal notice repeated on every page pulled out of a PDF. If that text lands in every chunk, it pushes all the vectors in the same direction and the distances between them shrink. The symptom is retrieval returning near-random results with suspiciously similar scores, and the cause is not chunk size: it is what you failed to strip before cutting.

The test set’s best size is not production’s. Measuring with thirty well-written questions, all answerable in one sentence, points at short chunks. Real user questions are vaguer and broader, and favour larger ones. If the test set does not look like the traffic, it optimises for the wrong system.

Context loss is real and has a known fix. Late chunking, proposed in 2024, sidesteps the problem by embedding the whole long text and applying the cut afterwards, before pooling — so every chunk vector carries the surrounding document with it⁵. Worth knowing that exists before you spend a week tuning sizes.

How to decide on your own corpus

1. Cut on structure wherever structure exists. Heading, section, clause, function, notebook cell. The structure was written by somebody who knew what belonged together.
2. Use fixed size only for what is left, with 300 tokens and 15% overlap as a starting point.
3. Prepend the heading path to the chunk’s text. A chunk that starts with “Pricing > Overage > Pro plan” retrieves better and gives the model the context the cut removed. It is the cheapest change on this list.
4. Never split a table without repeating the header in every fragment.
5. Change one variable at a time. Thirty real questions with the passage that should have been retrieved written down, and the metric being how often that passage lands in the top 5. Vary only the size, then only the strategy.
6. Run the test on a thousand documents, not the whole corpus. Having to reindex in order to test is what makes teams stop testing.

What changing your mind costs

Every chunking experiment is a reindex, because the chunk is the unit that became a vector. On a corpus of 3 million characters, roughly 900,000 tokens, each variant costs one full embedding pass — the cheapest call in the catalogue, but multiplied by however many variants you want to compare.

On the other side, the cost that repeats: five chunks of 300 tokens is 1,500 tokens of context per question; five chunks of 1,000 is 5,000. On a corpus serving a thousand questions a day, that gap is the whole argument for measuring before choosing rather than copying a default from a tutorial.

There is a hidden cost between the two, and it is usually the one that decides: storing the original text beside the vector, with each chunk’s offset inside its source file. That is a handful of bytes per chunk and it is what lets you change your mind without reindexing — expand the passage, merge neighbours, redo the cut from the document rather than from the database. An index that kept only the vector and the clipped text forces you to start over on every experiment, and that is how a chunking decision becomes permanent without anyone having decided it.

Footnotes

1. Jimeno Yepes et al. (2024) chunked financial reports by structural element type rather than by paragraph and reported improved RAG results, with a chunk size that comes out good without tuning. ↩

2. Qu et al. (2024) evaluated semantic chunking on document retrieval, evidence retrieval and answer generation, and concluded that its computational cost is not justified by consistent gains over fixed size. ↩

3. Duarte et al. (2024) used an LLM to find the point where content shifts and measured a 7.37% gain in DCG@20 over the strongest baseline, on a benchmark of a hundred public-domain books. ↩

4. Chen et al. (2023) compared indexing granularities and showed that units finer than a passage — propositions — beat passage-level indexing at retrieval and at downstream question answering. ↩

5. Günther et al. (2024) proposed embedding the whole long text and applying the cut after the transformer, before pooling, so that each chunk vector carries its surrounding context. ↩