Title: Syllabus
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation

AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation — Syllabus

Course description

This course prepares students to design, evaluate, and operate production-grade Retrieval-Augmented Generation (RAG) systems. Students will select and tune vector databases, build hybrid retrieval and reranking pipelines, and construct automated evaluation suites that gate quality in CI/CD. The course concludes with hands-on instrumentation of observability, prompt versioning, and drift-monitoring practices used to keep LLM applications reliable in production.

Contact hours

54 (3-unit equivalent), 16-week term, approximately 3.4 hours/week. Non-credit (CDCP Certificate of Completion pathway).

Prerequisite

AIINFRA 300 (Python + LLM app experience).

Course Learning Outcomes (CLOs)

1. CLO1: Select and configure a vector database (Qdrant, Milvus, Weaviate, or pgvector) based on scale, filtering, and indexing tradeoffs.
2. CLO2: Design chunking and hybrid retrieval pipelines combining dense embeddings, BM25, and cross-encoder reranking.
3. CLO3: Build automated RAG evaluation suites with RAGAS and DeepEval to gate retrieval and generation quality in CI/CD.
4. CLO4: Instrument LLM applications with tracing and observability platforms to diagnose latency, cost, and quality regressions.
5. CLO5: Implement production monitoring, prompt versioning, and feedback loops to detect and respond to model and data drift.

Weekly schedule

Week | Topic
01 | Introduction to Production RAG Architecture and the LLMOps Lifecycle
02 | Embeddings and Vector Representations for Retrieval
03 | Vector Database Selection: Qdrant, Milvus, Weaviate, and pgvector
04 | Indexing Deep Dive: HNSW vs IVF and Metadata Filtering
05 | Chunking Strategies for Real-World Documents
06 | Hybrid Search: Combining Dense Retrieval with BM25
07 | Cross-Encoder Rerankers and Query Rewriting/Expansion
08 | Advanced Retrieval Patterns: GraphRAG and Agentic RAG (Midterm)
09 | RAG Evaluation Foundations with RAGAS
10 | Deep Evaluation and Failure Analysis with DeepEval
11 | Wiring Evaluation into CI/CD Quality Gates
12 | Tracing and Observability: Langfuse and Arize Phoenix
13 | Observability at Scale: LangSmith, OpenLLMetry, and MLflow
14 | Prompt Management, Versioning, and Release Workflows
15 | Production Monitoring: Drift Detection, Feedback Loops, Cost, and Latency
16 | Capstone Project & Course Review (Capstone)

Grading

Component | Weight
Labs | 40%
Discussions | 10%
Weekly Quizzes | 15%
Midterm | 15%
Final Capstone | 20%

Credit/No-Credit. 70% overall is required to pass.

Course policies

Academic integrity: Students are expected to submit their own work. Collaboration on concepts is encouraged, but labs, quizzes, and the capstone must reflect individual understanding and effort. Submitting another student's work as your own, or misrepresenting AI-generated output as original analysis, is a violation of the LACCD Student Code of Conduct. Late work: Assignments are due as posted in Canvas. Late submissions are accepted up to 7 calendar days after the due date with no penalty, in recognition of the working-adult schedules typical of non-credit students; extensions beyond that window require instructor approval. Responsible use of AI: This course is built around production LLM systems, so use of AI tools (including LLM copilots) is permitted and expected for coding and drafting assistance. Students must be able to explain, defend, and debug any AI-assisted work they submit, and must disclose significant AI assistance on written reflections and the capstone. Accessibility: This course is committed to full inclusion of students with disabilities. Students needing accommodations should contact the campus Disabled Students Programs and Services (DSPS) office as early in the term as possible so reasonable accommodations can be arranged. All course materials are designed to meet WCAG 2.1 AA accessibility standards where feasible.

Tools & materials

All tools are free or free-tier for course use.

- Vector databases: Qdrant, Milvus, Weaviate, pgvector
- Embedding & retrieval: Sentence Transformers, OpenAI Embeddings API, rank_bm25, LangChain retrievers
- Rerankers & query enhancement: Cohere Rerank, cross-encoder models (Hugging Face), HyDE query expansion
- RAG evaluation: RAGAS, DeepEval
- LLM observability & tracing: Langfuse, Arize Phoenix, LangSmith, OpenLLMetry
- MLOps & experiment tracking: MLflow, Weights & Biases
- Low-code prototyping on-ramps: n8n, Langflow, Flowise

Week 01 · Introduction to Production RAG Architecture and the LLMOps Lifecycle

Notion ID: 392c08fd-0278-81bf-88c5-d27430f2a072 Notion URL: https://app.notion.com/p/392c08fd027881bf88c5d27430f2a072