Title: Quiz 10 — Evaluating Fine-Tuned Models and Detecting Overfitting
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 10 — Evaluating Fine-Tuned Models and Detecting Overfitting

Q: Perplexity is a reliable primary metric for evaluating how well an instruction-tuned chat model follows instructions.
- True
- False
A: False

Q: What is the classic signature of overfitting in a training run?
- Both training loss and validation loss decrease together and plateau at the same rate
- Training loss decreases while validation loss stops decreasing and then rises
- Validation loss decreases while training loss rises
- Both losses oscillate randomly with no trend
A: Training loss decreases while validation loss stops decreasing and then rises

Q: Which benchmark is most associated with checking for general-capability regression (forgetting) after fine-tuning, as opposed to checking task-specific gains?
- GSM8K
- HellaSwag
- Your custom training-adjacent holdout
- A domain-specific rubric you wrote yourself
A: HellaSwag

Q: Name two general-purpose benchmarks (besides HellaSwag) commonly used with lm-eval-harness to check for regression/forgetting after fine-tuning.
Feedback: Sample/expected answer: WinoGrande and ARC-Challenge

Q: Roughly what drop in general-benchmark score after fine-tuning is treated as a signal of possible catastrophic forgetting?
- 0.1–0.5 points
- \~2–3 points
- 10+ points
- Any drop at all, even 0.01 points
A: \~2–3 points

Q: A high score on a training-adjacent holdout set is sufficient proof that a fine-tune improved the model, without needing further regression testing.
- True
- False
A: False

Q: What does benchmark contamination refer to, and what is one of its effects?
- Corrupted model weights; causes random crashes during inference
- Unintended overlap between training data and evaluation benchmarks; can artificially inflate scores, sometimes increasing paradoxically as contamination increases
- Using the wrong tokenizer; causes evaluation scripts to fail to run
- Mixing quantized and unquantized weights; causes only speed regressions
A: Unintended overlap between training data and evaluation benchmarks; can artificially inflate scores, sometimes increasing paradoxically as contamination increases

Q: According to the recommended evaluation structure from this week's lecture, under what two conditions should you actually ship a fine-tuned model?
Feedback: Sample/expected answer: Regression (general benchmark) sets hold steady, AND at least one genuinely independent improvement set shows a statistically meaningful gain (not just improvement on a training-adjacent holdout).