Title: Discussion 10 — Writing Rubrics an LLM Judge Can't Wiggle Out Of
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 10 — Deep Evaluation and Failure Analysis with DeepEval

💬 Discussion 10 — Writing Rubrics an LLM Judge Can't Wiggle Out Of (20 pts)

A vague G-Eval criteria string like "is this a good answer?" can produce noticeably different scores across identical repeated runs. Why does that happen given that we're asking an LLM to "judge" the same fixed input every time, and what specifically makes a rubric tight enough to fix it? In your initial post (150+ words), address:

- Why "good" is an underspecified instruction for an LLM-as-judge, and what it means for the judge's chain-of-thought to be under-constrained.
- A before/after example: write one vague criteria string and rewrite it as a specific one for a RAG use case of your choice (yours or from the lab).
- Whether you'd expect your rewritten criteria to still show any run-to-run variance, and why (or why not).

Respond to at least two classmates — ideally push back on or extend their before/after rewrite with a scenario where it might still misfire.