Title: Midterm Exam
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 08 — Advanced Retrieval Patterns: GraphRAG and Agentic RAG

📝 Midterm Exam (100 pts)

Instructor note: the answer key below (marked with →) is hidden from students in the published Canvas quiz.

Part A — Applied Scenario (40 pts)

You've been hired as a contractor by a small nonprofit that runs a public health hotline. They have three years of call-summary transcripts (roughly 40,000 short documents) and want a system where staff can ask natural-language questions like "What's the standard guidance we give callers about flu symptoms?" and get an accurate, sourced answer. They have a modest budget and no dedicated MLOps team. Using what you've learned in Weeks 1–7, describe the RAG system you would build for them. Address at least: (1) how you'd chunk and embed the transcripts, (2) what vector database and index type you'd choose and why, (3) how you'd evaluate answer quality before launch, and (4) what you'd monitor once it's live in production. → A strong answer chunks transcripts by logical unit (e.g., per call or per Q&A turn) rather than fixed-size splitting, since call summaries have natural boundaries; uses a general-purpose embedding model appropriate to budget constraints; picks a vector database matching their scale and ops capacity (e.g., a managed/lightweight option given "no dedicated MLOps team," with an ANN index like HNSW for sub-second search at 40k-document scale); evaluates pre-launch with a labeled question/answer set measuring retrieval metrics (precision@k, recall@k) plus generation-quality metrics (faithfulness/groundedness, answer relevance), possibly using an LLM-as-judge pipeline; and monitors in production for latency, retrieval relevance drift, hallucination/faithfulness rate on live traffic, and user feedback signals (thumbs up/down), with logging/tracing so failures can be traced back to the specific retrieved chunks.

Part B — Weeks 1–7 Review (60 pts, 5 pts each)

1. (MC) In a standard RAG pipeline, what is the correct order of operations?

a) Generate → Retrieve → Embed b) Embed query → Retrieve → Generate c) Retrieve → Embed → Generate d) Generate → Embed → Retrieve → b

1. (TF) Larger chunk sizes always produce better retrieval results because they contain more context.

→ False — larger chunks can dilute relevance and exceed context budgets; the right chunk size depends on document structure and retrieval granularity needs.

1. (Short answer) What is the difference between sparse and dense retrieval?

→ Sparse retrieval (e.g., BM25) matches based on exact keyword/term overlap and term frequency statistics; dense retrieval uses learned vector embeddings to match on semantic similarity, catching paraphrases and related concepts that don't share exact words.

1. (MC) Which metric specifically measures whether a generated answer is supported by the retrieved context, rather than just being on-topic?

a) Recall@k b) Faithfulness/groundedness c) BLEU d) Latency → b

1. (TF) A vector database's ANN (approximate nearest neighbor) index trades a small amount of retrieval accuracy for significantly faster query speed at scale.

→ True

1. (Short answer) Name one cause of "context window overflow" in RAG and one mitigation.

→ Cause: retrieving too many/too-large chunks so the combined prompt exceeds the model's context limit. Mitigation: re-rank and truncate to the top-k most relevant chunks, summarize/compress retrieved context, or use a model with a larger context window.

1. (MC) Which of the following is an example of "LLM-as-judge" evaluation?

a) Computing cosine similarity between embeddings b) Using a second LLM to score a generated answer against a rubric c) Counting exact string matches d) Measuring server response time → b

1. (TF) Observability in a production LLM system is only about logging errors and stack traces, the same as in traditional software.

→ False — LLM observability also needs to capture prompts, retrieved context, token usage/cost, model outputs, and quality/faithfulness signals — not just crashes.

1. (Short answer) What is "re-ranking" in a RAG pipeline, and where does it sit in the process?

→ Re-ranking is a second-pass scoring step, typically using a cross-encoder or more expensive relevance model, applied to an initial set of candidate chunks retrieved by a fast first-pass method (e.g., vector similarity), to reorder them by true relevance before passing the top few to the generator.

1. (MC) A sudden drop in retrieval precision after a knowledge base update most likely indicates:

a) The LLM's temperature setting changed b) An embedding model mismatch or stale/incorrectly re-indexed vectors c) The user's internet connection is slow d) The vector database ran out of disk space → b

1. (TF) Hallucination can still occur in RAG systems even when relevant context is correctly retrieved.

→ True — the generator can still ignore, misread, or embellish beyond the provided context, which is why faithfulness checks are needed even with good retrieval.

1. (Short answer) Why is it important to version and log both the prompt template and the retrieved context for each production query, not just the final answer?

→ Because when quality issues appear later, you need to reproduce and debug the exact conditions that produced a bad answer — without the prompt version and the specific retrieved context, you can't tell whether the failure was a retrieval problem, a prompt problem, or a model problem, and you can't reliably re-test a fix.