Title: Lab 5 — Recursive vs. Semantic Chunking Showdown
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 05 — Chunking Strategies for Real-World Documents

🛠️ Lab 5 — Recursive vs. Semantic Chunking Showdown (50 pts)

Goal: Using free, open-source LangChain text splitters, compare RecursiveCharacterTextSplitter against the embedding-driven SemanticChunker on a messy, real-world document. You'll measure chunk count, boundary quality, and downstream answer relevancy — all with local, no-cost tools. Steps:

1. Pick a messy real-world document. Choose one PDF or HTML page that has real structure to break: a syllabus, a government form, a product manual, or a Wikipedia article with tables and headers. Avoid clean plain-text files — the point is to stress-test structure handling.
2. Set up locally, no paid APIs. Install langchain, langchain-experimental, langchain-community, and sentence-transformers in a Python environment (venv or Conda). Use a local embedding model such as sentence-transformers/all-MiniLM-L6-v2 for both chunking and retrieval — no OpenAI or other paid API keys required.
3. Load and extract the document. Use a free loader (e.g., PyPDFLoader or UnstructuredHTMLLoader from langchain-community) to pull the raw text out of your chosen file. Note anything that looks obviously mis-extracted (garbled tables, missing headers) before you even start chunking — that's parsing quality, and it matters more than your chunker choice.
4. Chunk with RecursiveCharacterTextSplitter. Configure it with chunk_size=400 and chunk_overlap=60 (roughly 15% overlap) as your starting point. Run it on your document and save the resulting chunks.
5. Chunk with SemanticChunker. Import SemanticChunker from langchain_experimental.text_splitter, pass it your local sentence-transformers embedding model, and run it on the same document. Save this second set of chunks.
6. Compare boundary quality by hand. For each chunk set, manually inspect at least 10 chunks per method. Record: chunk count, average chunk length, and how many chunks cut mid-sentence or split a table/list awkwardly. Build a small table of your findings.
7. Build a shared local retrieval pipeline. Embed both chunk sets with the same sentence-transformers model into a local vector store (FAISS or Chroma, both free and local). Write 5 realistic questions a user might ask about your document.
8. Run retrieval and judge relevancy. For each question, retrieve the top-3 chunks from each chunk set and judge (yourself, using your own reading) whether the retrieved chunks actually contain a complete, usable answer. Score each method 1–5 per question for answer relevancy.
9. Write up your findings. Summarize which method produced better boundaries, which produced better retrieval, whether the extra cost of semantic chunking was worth it for this document, and what you'd change if this were a different type of corpus (e.g., Slack messages instead of a manual).

Deliverables: Submit your Python script or notebook, your comparison table (chunk count, avg length, mid-sentence/mid-table cuts per method), your 5 questions with per-method relevancy scores, and a short written recommendation (200–300 words) on which chunking strategy you'd use for this document and why.