Title: Final Capstone Project
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 16 — Capstone Project & Course Review

🎓 Final Capstone Project (100 pts)

The scenario

Pick (or continue with) a real or realistic use case for a retrieval-augmented generation system — something you'd genuinely want to run in production. Examples: an internal policy/HR assistant, a customer-support knowledge base bot, a technical documentation Q&A tool, a research-paper assistant, or a domain-specific assistant for your own job or industry. You will design the full production system and build a working prototype of its core pieces — you are not expected to build a polished, fully deployed product, but your prototype must actually run and produce evidence (traces, eval scores, a dashboard view, etc.), not just describe what it would do.

Requirements

Your capstone must include all three of the following: 1. Design document (written) A written document that maps your design decisions explicitly back to the skills from this course. At minimum, address:

1. System overview — what the assistant does, who uses it, and what "success" looks like for it.
2. RAG architecture (Weeks 1–3) — your chunking strategy, embedding model choice, and vector database choice, with your reasoning.
3. Retrieval quality (Weeks 4–6) — how you improve retrieval beyond naive top-k similarity search (hybrid search, re-ranking, metadata filtering, etc.) and why.
4. LLMOps foundations (Weeks 7–9) — how you version prompts, trace requests, and track cost/latency in this system.
5. Evaluation strategy (Weeks 10–12) — your golden dataset (what's in it and how you built it), your automated eval pipeline, and where/how you use LLM-as-judge.
6. Observability plan (Weeks 13–15) — what you'd put on a production dashboard, what drift or failure modes you'd watch for, and what your alerting/incident-response plan would be.
7. Known limitations and next steps — what you did NOT build, and what you'd tackle first if given another month.

2. Diagram A single diagram (architecture/flow diagram) showing how a user query moves through your system — ingestion → retrieval → generation → evaluation/observability layer. Any tool is fine (Excalidraw, draw.io, Mermaid, Lucidchart, even a clearly labeled hand-drawn diagram photographed or scanned). Embed it in your design document or submit it as a separate image/PDF alongside your write-up. 3. Working prototype artifact A prototype, built using the free/low-cost toolchain from this course, that demonstrably runs. It must include at minimum:

- A retrieval pipeline over a real (even if small) document set
- At least one form of tracing or logging of requests through the pipeline
- A golden dataset (10+ question/answer pairs is enough) and the results of running your eval pipeline against it

Submit your prototype as a code repository link, notebook export, or zipped project folder, plus a short README explaining how to run it and what the included evidence (trace logs, eval output, screenshots of a dashboard, etc.) shows.

Deliverables

- [ ] Design document (Word/PDF/Markdown) addressing all 7 points above
- [ ] Architecture/flow diagram (embedded or attached separately)
- [ ] Working prototype artifact (repo link, notebook, or zipped folder) with a README
- [ ] Evidence from your prototype run: at least one trace/log example and your eval pipeline's output against your golden dataset

Grading

Graded out of 100 points using the Capstone / Project Rubric. This assignment maps to all five course learning outcomes (CLO1–CLO5) — it is your chance to show what you can do with everything from this course in one connected system.