Title: Quiz 14 — Knowledge Distillation and Quality-Size Tradeoff Analysis
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 14 — Knowledge Distillation and Quality-Size Tradeoff Analysis

Q: In knowledge distillation terminology, the smaller model being trained to imitate a larger model is called the:
- Teacher
- Student
- Oracle
- Baseline
A: Student

Q: You only have API access to a closed commercial model and want to distill it into your own model. Which type of distillation are you limited to?
- White-box KD
- Reverse-mode KD
- Black-box KD
- On-policy KD
A: Black-box KD

Q: White-box knowledge distillation requires access to the teacher model's internal parameters, such as its full output logits/probability distribution, not just its final text output.
- True
- False
A: True

Q: MiniLLM proposes using which objective for distilling generative language models, instead of standard forward KL-divergence?
- Mean squared error on embeddings
- Reverse KL-divergence
- Cosine similarity loss
- Contrastive loss
A: Reverse KL-divergence

Q: What does "Distilling Step-by-Step" extract from the teacher model that a standard answer-only distillation approach does not use?
Feedback: Sample/expected answer: Chain-of-thought rationales (the teacher's reasoning steps), which are used alongside final answers to train a more data-efficient student.

Q: TRL's GKDTrainer addresses the distribution-mismatch problem in autoregressive distillation primarily by:
- Increasing the student's learning rate
- Training the student on its own self-generated sequences with teacher feedback (on-policy training)
- Freezing all student weights except the output layer
- Only ever training on teacher-authored text
A: Training the student on its own self-generated sequences with teacher feedback (on-policy training)

Q: A distilled student model is guaranteed to outperform a directly-trained (non-distilled) baseline of the same parameter count.
- True
- False
A: False

Q: The core motivation for distilling large LLMs into smaller ones is best described as:
- Improving the teacher model's own accuracy
- Reducing inference cost while approximating the teacher's quality
- Eliminating the need for any evaluation of the student
- Making the student model larger than the teacher
A: Reducing inference cost while approximating the teacher's quality