Chunking Strategies Compared: Fixed, Recursive, Semantic and Late Chunking

What a chunker is actually deciding

A chunker turns a document into retrieval units. Each unit is embedded once, stored in the vector index, and returned whole when it ranks in the top k. Three things follow from that. The unit is the smallest thing retrieval can find, so if the answer is cut across two units, both must rank in the top k for the model to see it. The unit's embedding is a pooled summary of everything inside it, so a sentence that only makes sense with its neighbours ("It ships within three days") is represented without them. And every retrieved unit is pasted into the prompt, so unit size times k is a direct, per-request cost.

A useful way to evaluate a chunker is therefore not "are the chunks pretty" but three numbers on a set of real questions with known answer spans: the share of answers that are split across units, answerable recall@k (the retrieved units together contain the whole answer span), and context tokens per query. The lab reports exactly these, plus index size and the embedding overhead introduced by overlap.

Fixed-size windows and the arithmetic of cuts

The simplest chunker slides a window of S tokens across the text with a stride of S - O, where O is the overlap. It is fast, predictable and blind to structure. The probability that it cuts a given answer is easy to estimate: an answer span of length L survives intact only if no boundary falls inside it. With boundaries every S - O tokens and a random offset, the chance of a cut is roughly min(1, (L - 1 - O) / (S - O)) when L > O, and zero otherwise.

# split probability for a 60-token answer (two sentences), calculated
S = 128, O = 0   -> 59 / 128 = 46% of such answers are cut
S = 256, O = 0   -> 59 / 256 = 23%
S = 256, O = 64  -> (59 - 64) < 0, so the span always fits in some window
# embedding overhead of overlap = S / (S - O) - 1
S = 256, O = 64  -> 256 / 192 - 1 = 33% more tokens embedded and stored

Overlap therefore buys protection for answers shorter than the overlap, at a linear cost in index size and embedding spend, and it also returns duplicated text in the context when adjacent windows both rank. In the lab, fixed 128-token windows split about a fifth of answers and reach only about 52% answerable recall@3; doubling the window to 256 lifts recall to about 72% but nearly doubles context tokens.

Recursive splitting: let the document choose the boundaries

Recursive splitters try the coarsest separator first and fall back to finer ones only when a piece is still too large: section breaks, then paragraph breaks, then sentence or line breaks, then characters. LangChain's RecursiveCharacterTextSplitter works this way with a separator list that starts with blank lines, and LlamaIndex's sentence splitter keeps sentences whole while packing them up to a token budget. Pieces are then merged greedily up to the size limit, never crossing a higher-level boundary.

The effect on splits is dramatic because most answers live inside one paragraph. In the lab, recursive splitting at 256 tokens cuts essentially no answers and reaches about 74% recall@3 with about 540 context tokens per query, against 72% and 690 tokens for fixed windows of the same maximum size. It wins on both axes because its chunks are on average smaller than the limit (it stops at boundaries) and coherent. Two cautions: units are measured in characters by default in some splitters and in tokens in others, so a "chunk_size of 1000" can mean very different things; and documents converted from PDF often lose their structure, so the recursive splitter silently degrades into a fixed one. Check your loader output before trusting the separator list, and check the defaults for the library version you run.

Semantic chunking: breakpoints from embeddings

Semantic chunkers embed each sentence, compute the similarity between neighbours, and start a new chunk where similarity drops, on the theory that topic shifts are where the meaning changes. LangChain's experimental SemanticChunker picks breakpoints from a percentile, standard deviation, interquartile or gradient rule over the distances; the lab uses a simpler absolute threshold so the mechanism is visible. The appeal is real: chunks follow content rather than formatting, and repeated boilerplate tends to end up isolated in its own chunk instead of polluting every neighbour. In the lab, adding a disclaimer sentence to every paragraph costs fixed windows about nine points of recall while the semantic chunker actually gains, because the disclaimers become their own chunks.

The weakness is that sentence embeddings are a noisy signal for topic boundaries. A sentence that refers back with a pronoun has a standalone embedding that looks unlike its predecessor, so the chunker breaks exactly where context is most needed. With an aggressive threshold the result is near one-sentence chunks: cheap in context tokens, but multi-sentence answers are scattered. In the lab with multi-sentence answers, a threshold of 0.4 leaves about a third of answers split and recall near 51%. Semantic chunking also costs an embedding call per sentence at index time. Published comparisons have found its advantage over well-tuned fixed or structural splitting to be inconsistent across datasets, so treat it as a hypothesis to test, not an upgrade.

Late chunking: embed first, split second

Every strategy above embeds each chunk in isolation. Late chunking, described by Günther and colleagues at Jina AI in 2024, reverses the order: run a long-context embedding model over the whole document (or as much as fits in its window), keep the per-token output vectors, and only then mean-pool the token vectors that fall inside each chunk's boundaries. Because every token's vector was computed with attention over the full document, the tokens of "It ships within three days" already carry information about which product "it" is.

Late chunking is orthogonal to how you choose boundaries: you can apply it to fixed, recursive or semantic chunks, which is why the lab exposes it as a toggle. Its cost is that the embedding model must accept the whole document; documents longer than the model's window are processed in macro-chunks and lose cross-window context. You also need either a model and API that support it (Jina's embedding API exposes a late chunking option on recent models; check the current documentation) or access to token-level outputs of an open model. In the lab's pronoun-heavy corpus, recursive 256-token chunks score about 58% recall@3; turning on late chunking lifts the same boundaries to about 81% at the same context cost. An alternative with a similar goal is to prepend a short, generated description of the document to each chunk before embedding, as in Anthropic's contextual retrieval write-up; it costs an LLM call per chunk at index time instead of a long-context embedding.

Chunk size, top-k and the context bill

Once boundaries respect structure, the remaining dial is size, and it trades recall against context tokens almost linearly. Context tokens per query are roughly k × average chunk tokens. At one million queries a month, each extra 1,000 input tokens per query is a billion tokens a month; at an illustrative $1 per million input tokens that is $1,000 a month before any output. Large chunks also make the LLM's job harder: answers buried in the middle of long contexts are used less reliably, a finding reported in the "Lost in the Middle" study.

In this synthetic model recall keeps rising with size because a few huge chunks cover most of a document, so the honest comparison is recall per context token, not recall alone. The fourth row is the interesting one: it matches the 512-token fixed setup's recall at well under half its context cost. Real corpora add a second penalty for large chunks that the lab does not model: a pooled embedding of many unrelated sentences matches specific questions less sharply.

A common way out of the size dilemma is to decouple the unit you search from the unit you return. Small-to-big patterns index small chunks (a sentence or a few sentences) for sharp matching, then expand each hit to its parent paragraph or section, or to a window of neighbouring sentences, before building the prompt. LangChain's parent document retriever and LlamaIndex's sentence-window and auto-merging retrievers implement variants of this. The cost moves rather than disappears: you store both levels, and the expanded context still has to fit the token budget, so cap the expansion and deduplicate overlapping parents.

Worked example: re-chunking a catalog knowledge base

Suppose the hypothetical Belgavi's Book Store keeps 40,000 product and policy pages, averaging 900 tokens, chunked with fixed 512-token windows and top-3 retrieval. That is about 70,000 chunks and 1,250 context tokens per query (the lab's figure for this setup). Moving to recursive 256-token chunks with late chunking roughly doubles the chunk count to about 160,000, which at 1,024 dimensions in float32 is about 650 MB of raw vectors instead of about 290 MB, and cuts context to about 540 tokens per query. At 2 million queries a month that removes roughly 1.4 billion input tokens. Re-embedding 36 million tokens once is cheap by comparison. All figures are calculated from the stated assumptions and the lab's synthetic results, not measured.

Before re-indexing anything, build an evaluation set: a few hundred real questions, each labelled with the document and the character span that answers it. Then run each candidate chunker, retrieve, and compute split rate, answerable recall@k and context tokens offline. Re-indexing a large corpus is slow and expensive; a span-labelled evaluation set makes it a measured decision rather than a hopeful one.

def answerable(retrieved, span):
    # retrieved: list of (doc_id, start, end) character or token ranges
    cover = span.start
    for doc, a, b in sorted(r for r in retrieved if r[0] == span.doc):
        if a <= cover < b:
            cover = b
    return cover >= span.end

Failure modes and how to spot them in production

• Split answers. The model answers half a procedure or omits a condition. Detect it by logging retrieved chunk IDs and checking how often the cited chunk ends mid-list or mid-table; in evaluation, track split rate directly.
• Orphaned references. Retrieval returns chunks about the right topic but the wrong product, because the right chunk only says "it". Late chunking, contextual prefixes or adding the title to every chunk all address this.
• Structure lost in conversion. PDF and HTML extraction flatten headings and tables, and recursive splitters fall back to character cuts. Inspect a sample of chunks from every source type after each loader upgrade.
• Tables and code. Row-by-row splitting separates values from headers. Keep tables whole, or repeat the header row in each piece.
• Silent cost growth. Raising chunk size or k to fix one bad answer increases every request's bill. Put context tokens per query on the same dashboard as answer quality.

Common mistakes: comparing chunkers by eyeballing a few chunks, measuring retrieval hit rate by document instead of by answer span, mixing character and token units, and re-indexing without versioning the chunker configuration alongside the index.

How the lab maps to these decisions

The four missions walk through the decisions in order. Mission one starts from fixed 128-token windows and asks you to stop splitting answers without blowing the context budget, which teaches the value of structural boundaries over overlap. Mission two turns on pronoun-heavy documents and asks for 80% recall within 800 tokens, which bigger chunks cannot deliver but late chunking can. Mission three starts from whole sections with multi-sentence answers and asks you to keep 85% recall under 1,100 tokens, a genuine trade between size and k. Mission four keeps the semantic chunker and asks you to make it work on multi-sentence answers by breaking less often and adding context.

Run the lab: Chunking showdown: keep answers whole without flooding the context

The lab below builds a synthetic corpus of 28 product documents with sections, paragraphs and sentences, chunks it with the strategy you pick, embeds every chunk, and retrieves the top k chunks for 260 generated questions. The strips at the top show two documents: coloured blocks are sentences by section, faded blocks are sentences that say "it" instead of naming the product, and white ticks mark chunk boundaries. A highlighted box cycles through questions, green when the retrieved chunks contain the whole answer and red when they do not. The chart shows recall for the same strategy at every chunk size.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Label 200 to 500 real questions with the document and exact span that answers them, and keep that set under version control.
2. Compute split rate, answerable recall@k and context tokens per query for your current chunker before changing anything.
3. Check whether your splitter measures chunk size in characters or tokens in the version you run, and record the unit next to the value.
4. Inspect 50 chunks from each source type (PDF, HTML, Markdown) for lost headings, broken tables and orphaned pronouns.
5. Try recursive splitting on real structure and late chunking or contextual prefixes on the same boundaries, and compare on the labelled set.
6. Version the chunker configuration together with the index, and add context tokens per query to the RAG quality dashboard.

Sources and further reading

• Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (Günther et al., 2024)
• Is Semantic Chunking Worth the Computational Cost? (Qu, Bao and Tu, 2024)
• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., TACL 2024)
• LangChain documentation: text splitters
• Anthropic: Introducing Contextual Retrieval

Related in this series:

• HNSW Tuning for RAG: M, efSearch and the Recall Versus Latency Trade
• Lost in the Middle: Ordering and Packing Retrieved Context for Long Prompts
• RAG Failure Triage: Separating Retrieval Misses From Generation Errors

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.