Title: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation

00 · Course Information
  Outcomes & Rubrics
Week 01 — Introduction to Production RAG Architecture and the LLMOps Lifecycle
  Lecture 1 — Why Production RAG Needs an Operations Lifecycle
  Lab 1 — Instrument a Mini RAG Function with Local MLflow Tracing
  Discussion 1 — Where MLOps Ends and LLMOps Begins
  Quiz 1 — Introduction to Production RAG Architecture and the LLMOps Lifecycle
  Week 01 Resources
Week 02 — Embeddings and Vector Representations for Retrieval
  Lecture 2 — From Text to Vectors: Embeddings, Similarity, and the 2026 Model Landscape
  Lab 2 — Benchmark Two Embedding Models with MTEB
  Discussion 2 — Building a Defensible Embedding-Selection Process
  Quiz 2 — Embeddings and Vector Representations for Retrieval
  Week 02 Resources
Week 03 — Vector Database Selection: Qdrant, Milvus, Weaviate, and pgvector
  Lecture 3 — Choosing a Vector Database Without the Hype
  Lab 3 — Four Engines, One Dataset, One Comparison
  Discussion 3 — Picking Your Engine
  Quiz 3 — Vector Database Selection: Qdrant, Milvus, Weaviate, and pgvector
  Week 03 Resources
Week 04 — Indexing Deep Dive: HNSW vs IVF and Metadata Filtering
  Lecture 4 — Inside the Index: HNSW, IVF, and the Filtering Problem
  Lab 4 — Build, Benchmark, and Filter: HNSW vs IVF Head-to-Head
  Discussion 4 — Choosing an Index When Filtering Dominates Your Workload
  Quiz 4 — Indexing Deep Dive: HNSW vs IVF and Metadata Filtering
  Week 04 Resources
Week 05 — Chunking Strategies for Real-World Documents
  Lecture 5 — Chunking Strategies for Real-World Documents
  Lab 5 — Recursive vs. Semantic Chunking Showdown
  Discussion 5 — One Hyperparameter to Rule Them All?
  Quiz 5 — Chunking Strategies for Real-World Documents
  Week 05 Resources
Week 06 — Hybrid Search: Combining Dense Retrieval with BM25
  Lecture 6 — Why Hybrid Search Wins: Fusing BM25 and Dense Retrieval with RRF
  Lab 6 — Breaking and Fixing Retrieval with Hybrid Search and RRF
  Discussion 6 — Tuning Fusion for Your Query Mix
  Quiz 6 — Hybrid Search: Combining Dense Retrieval with BM25
  Week 06 Resources
Week 07 — Cross-Encoder Rerankers and Query Rewriting/Expansion
  Lecture 7 — Reranking and Rewriting: Getting the Right Documents to the Top
  Lab 7 — Retrieve, Rerank, and Rewrite: Measuring the Lift
  Discussion 7 — What Reranking Alone Can't Fix
  Quiz 7 — Cross-Encoder Rerankers and Query Rewriting/Expansion
  Week 07 Resources
Week 08 — Advanced Retrieval Patterns: GraphRAG and Agentic RAG
  Lecture 8 — From Fixed Pipelines to Graphs and Agents
  Midterm Exam
  Discussion 8 — Graph vs. Agent: Picking Your Battles
  Week 08 Resources
Week 09 — RAG Evaluation Foundations with RAGAS
  Lecture 9 — Judging RAG Without Judging Everything by Hand
  Lab 9 — Scoring Your RAG Pipeline with RAGAS
  Discussion 9 — Faithful but Incomplete
  Quiz 9 — RAG Evaluation Foundations with RAGAS
  Week 09 Resources
Week 10 — Deep Evaluation and Failure Analysis with DeepEval
  Lecture 10 — From Vibes to Verdicts: Unit Testing LLM Apps with DeepEval
  Lab 10 — Catching Failures with DeepEval Metrics
  Discussion 10 — Writing Rubrics an LLM Judge Can't Wiggle Out Of
  Quiz 10 — Deep Evaluation and Failure Analysis with DeepEval
  Week 10 Resources
Week 11 — Wiring Evaluation into CI/CD Quality Gates
  Lecture 11 — From Manual Evals to Automated Quality Gates
  Lab 11 — Building a Merge-Blocking DeepEval Pipeline in GitHub Actions
  Discussion 11 — Designing Tiered Quality Gates
  Quiz 11 — Wiring Evaluation into CI/CD Quality Gates
  Week 11 Resources
Week 12 — Tracing and Observability: Langfuse and Arize Phoenix
  Lecture 12 — From Log Lines to Traces: How Langfuse and Phoenix See Your LLM App
  Lab 12 — Dual Tracing: Instrumenting a RAG App with Phoenix and Langfuse
  Discussion 12 — Why "Generation" Deserves Its Own Span Type
  Quiz 12 — Tracing and Observability: Langfuse and Arize Phoenix
  Week 12 Resources
Week 13 — Observability at Scale: LangSmith, OpenLLMetry, and MLflow
  Lecture 13 — Choosing (and Combining) Your Observability Stack
  Lab 13 — Three-Way Observability Shootout
  Discussion 13 — The 10x Trace Volume Problem
  Quiz 13 — Observability at Scale: LangSmith, OpenLLMetry, and MLflow
  Week 13 Resources
Week 14 — Prompt Management, Versioning, and Release Workflows
  Lecture 14 — From Loose Strings to a Prompt Registry
  Lab 14 — Version, Label, and Roll Back a Production Prompt
  Discussion 14 — Why Immutability Matters for Incident Debugging
  Quiz 14 — Prompt Management, Versioning, and Release Workflows
  Week 14 Resources
Week 15 — Production Monitoring: Drift Detection, Feedback Loops, Cost, and Latency
  Lecture 15 — Watching a RAG System That Never Stops Talking
  Lab 15 — Drift Reports and a Production Monitoring View
  Discussion 15 — The Dashboards Are Green, So Why Is the Chatbot Getting Worse?
  Quiz 15 — Production Monitoring: Drift Detection, Feedback Loops, Cost, and Latency
  Week 15 Resources
Week 16 — Capstone Project & Course Review
  Final Capstone Project
  Capstone Presentation
  Course Review & Reflection
  What's Next