Title: Lecture 5 — Chunking Strategies for Real-World Documents
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 05 — Chunking Strategies for Real-World Documents

📖 Lecture — Chunking Strategies for Real-World Documents

Every RAG pipeline you've built so far has quietly depended on one decision you probably didn't think much about: how the document got cut into pieces before it was embedded. This week we make that decision explicit. Chunking sounds like a minor preprocessing step, but it's one of the highest-leverage design choices in a RAG system — get it wrong and no amount of clever prompting or reranking will save you, because the retriever can only return chunks that actually contain a clean, complete answer. The three baseline strategies Fixed-size chunking is the simplest approach: pick a number of characters or tokens (say, 500) and cut the document into blocks of that size, usually with some overlap. It's fast, predictable, and completely blind to the document's structure. It will happily slice a sentence in half, split a table row from its header, or sever a bullet list from the paragraph that introduces it. It ignores the free structural signal — paragraph breaks, sentence boundaries, headers — that real business documents already carry. Recursive chunking, implemented as RecursiveCharacterTextSplitter in LangChain (and the equivalent default splitter in LlamaIndex), fixes much of this by trying a hierarchy of separators in order: first split on double newlines ("\n\n", paragraph breaks), and only if a resulting piece is still too large, fall back to single newlines ("\n"), then spaces (" "), and finally raw characters ("") as a last resort. The effect is that it keeps paragraphs and sentences intact whenever the size budget allows it, and only cuts mid-sentence when it truly has no other choice. This is why recursive chunking remains the default, general-purpose recommendation in both LangChain and LlamaIndex in 2026 — it's the safe starting point for the vast majority of RAG projects, and you should reach for something else only when you have a specific reason to. Semantic chunking goes a step further by using embeddings instead of separators to decide where to cut. LlamaIndex's SemanticSplitterNodeParser, for example, embeds consecutive sentences and measures the similarity between each pair. When the similarity drops by more than a set threshold (a percentile cutoff — 95th by default), that's treated as a topic shift, and a chunk boundary is inserted there. The result is chunks that are more likely to be internally coherent — one topic per chunk — because the boundary was chosen based on meaning, not on where a newline happened to fall. The cost is real: you now need an embedding call for every sentence at ingestion time, and the results can be less predictable in size. Reserve semantic chunking for corpora where topic coherence really matters and structure is unreliable — research papers, legal filings, dense technical documentation — rather than reaching for it everywhere by default. Starting numbers, not universal truths A reasonable starting point for most business documents is chunks of roughly 200–500 tokens with about 10–20% overlap between consecutive chunks. Overlap helps prevent an answer-bearing sentence from being orphaned right at a chunk boundary. But treat these as tunable defaults you validate against your own corpus and your own evaluation set — not constants you set once and forget.

Strategy | How it decides boundaries | Strengths | Watch out for
Fixed-size | Character/token count only | Fast, simple, predictable size | Ignores structure; cuts sentences, tables, headers
Recursive | Separator hierarchy (\n\n → \n → → "") | Respects paragraphs/sentences when possible; safe default | Still size-driven; can miss semantic topic shifts
Semantic | Embedding similarity between sentences, split at similarity drops | Coherent, single-topic chunks | Costs embedding calls at ingest; size less predictable; best saved for research/legal/technical docs

Correcting three common misconceptions First: "Just split every document into fixed 512-token chunks with 50-token overlap like the tutorials show." This is exactly the fixed-size approach above, and it throws away the structural signal your documents give you for free — sentence endings, paragraph breaks, table boundaries, and headers. Recursive chunking gets you most of the benefit with almost none of the extra cost, so there's rarely a reason to default to naive fixed-size splitting on real documents. Second: "There is one optimal chunk size, and finding it is the key to good RAG." Chunk size is corpus-dependent — what works for a Slack export won't work for a 40-page legal contract. Some controlled studies have even found naive fixed-size chunking outperform more elaborate context-aware chunking on certain datasets. The bigger levers are usually parsing correctness (did you extract the table and header text properly in the first place?) and evaluating on your own data, rather than obsessively grid-searching one chunk-size number. Third: "Bigger chunks are always safer because they include more context." This misunderstands the tradeoff as one-sided. Chunking is genuinely two-sided: chunks that are too small lose context and can strand an answer without its supporting detail; chunks that are too large dilute the embedding by mixing multiple topics into a single vector, so that vector ends up matching nothing particularly well at retrieval time. There's no free lunch in either direction — you're always balancing lost context against a muddied embedding. Where the field is heading Looking at 2026 guidance, the biggest gains in retrieval quality are increasingly coming from techniques like contextual retrieval (prepending a short LLM-generated summary of surrounding context to each chunk before embedding it) and late chunking (embedding the full document first, then deriving chunk-level vectors from those contextualized embeddings), rather than from further tuning of chunk size and overlap. In other words, chunking strategy has diminishing returns past a certain point — once you've picked a sensible strategy (recursive, by default) and reasonable starting numbers, your energy is often better spent on these newer techniques, on parsing quality, and on evaluation, than on squeezing one more percent out of your chunk-size hyperparameter.