Title: Lecture 9 — Judging RAG Without Judging Everything by Hand
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 09 — RAG Evaluation Foundations with RAGAS

📖 Lecture — Judging RAG Without Judging Everything by Hand

Up to this point in the program, you have built RAG pipelines and trusted your own eyeballs to tell you whether they were any good. That does not scale. Once a RAG system has more than a handful of test questions, or once it is running in production against real user traffic, you need a way to score quality automatically, repeatedly, and cheaply. This week introduces RAGAS (Retrieval-Augmented Generation Assessment), the most widely adopted open-source framework for exactly that job. The core idea: an LLM as judge. RAGAS does not require you to hand-label thousands of examples with human relevance judgments. Instead, it uses an LLM — the same kind of model you are already building applications with — as an automated judge. For every (question, retrieved context, generated answer) triple your pipeline produces, RAGAS prompts a judge LLM with carefully designed evaluation instructions and gets back a score between 0 and 1 for each metric. This is dramatically cheaper than paying human annotators to read context passages and answers one at a time, and it can run continuously in a CI pipeline or nightly evaluation job. The tradeoff is that your evaluation quality is now bounded by your judge model's judgment quality, which is why choosing and configuring the judge LLM deliberately matters. The four core metrics, split by component. The single most important mental model for this week is that RAGAS metrics are not one undifferentiated "RAG quality score" — they cleanly separate the retriever's job from the generator's job. If you only remember one table from this course, make it this one:

Metric | Pipeline stage | What it measures | Needs a reference answer?
Context Precision | Retriever | Fraction of retrieved chunks that are actually relevant, weighted so higher-ranked relevant chunks count more (order-sensitive) | Yes
Context Recall | Retriever | Whether the retrieved context contains everything needed to construct the ground-truth answer | Yes
Faithfulness | Generator | Whether every claim in the generated answer can be traced back to (inferred from) the retrieved context | No
Answer Relevancy | Generator | How directly the answer addresses the original question, independent of whether it is correct | No

Notice that Context Precision and Context Recall both require a reference ("ground truth") answer to compute, while Faithfulness and Answer Relevancy are reference-free — they only need the question, the retrieved context, and the generated answer. That is a practical distinction: reference-free metrics can run on live production traffic where you have no ground truth, while reference-based metrics require a curated evaluation set. How Answer Relevancy actually works. This metric surprises people the first time they see the mechanism. RAGAS does not ask the judge LLM "is this answer relevant?" and take its word for it. Instead, it takes the generated answer and asks the judge LLM to generate several hypothetical questions that the answer would be a good response to. It then embeds those hypothetical questions and the original question, and averages the cosine similarity between them. If the hypothetical questions land close to the original question, the answer was on-topic; if they drift away, the answer was padded, evasive, or off-topic. This is a clever indirect measurement, and it is the reason Answer Relevancy penalizes verbose, hedge-everything answers even when those answers are technically not wrong. Running RAGAS in practice. The quickstart flow is intentionally lightweight: pip install ragas (or uvx ragas quickstart rag_eval to scaffold a starter project), configure your judge LLM by setting the appropriate API key as an environment variable, and then run your evaluation script (commonly python evals.py) against a dataset of questions, contexts, and answers. RAGAS supports multiple judge LLM providers through a pattern called llm_factory — for example, you can point RAGAS at Claude as the judge LLM rather than defaulting to OpenAI, which matters if your organization has standardized on Anthropic models or wants to avoid mixing vendors between the system under test and the judge. Correcting three common misreadings of these scores. First: a high Faithfulness score does not mean the answer is factually correct. Faithfulness only checks whether the claims in the answer can be inferred from the retrieved context — if your retriever hands the generator an outdated or simply wrong passage, and the generator faithfully summarizes that wrong passage, Faithfulness will still score high. Faithfulness measures groundedness in your context, not truth in the world. Second: Answer Relevancy does not tell you whether the answer is right either — it measures on-topic-ness by the hypothetical-question mechanism described above, and explicitly does not evaluate factual accuracy. A confidently wrong but on-topic answer can still score well on relevancy. Third, and most important for debugging: Context Recall and Context Precision are not interchangeable measures of "retrieval quality." Recall tells you whether the needed information was retrieved at all, anywhere in the returned set. Precision tells you whether the chunks you retrieved were ranked well — whether relevant chunks floated to the top instead of getting buried under noise. A system can retrieve all the right information (high recall) but rank it so poorly that precision suffers, or it can retrieve a tight, well-ranked set (high precision) that is still missing a key fact (low recall). Conflating the two hides which half of your retriever — the search step or the ranking step — is actually the bottleneck, which is exactly the diagnostic skill this week's lab and discussion are built to practice.