Title: Quiz 9 — Preference Tuning: DPO in Practice (with a Look at GRPO)
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 09 — Preference Tuning: DPO in Practice (with a Look at GRPO)

Q: What does DPO primarily replace in the classic RLHF pipeline?
- The tokenizer
- The separate reward-model-training stage plus the PPO reinforcement learning loop
- The base model's architecture
- The evaluation benchmark
A: The separate reward-model-training stage plus the PPO reinforcement learning loop

Q: PPO-based RLHF requires four models in memory at once (policy, reference, reward, value function), while DPO requires only two (policy and reference).
- True
- False
A: True

Q: What role does the frozen reference model play in DPO training?
Feedback: Sample/expected answer: It provides a soft KL-divergence constraint that keeps the trained policy from drifting too far from its original behavior; mishandling it can destabilize training.

Q: Which statement about DPO's reward modeling is accurate?
- DPO has no reward model of any kind — it is pure supervised learning
- DPO still performs reward modeling, but implicitly, reparameterized directly from the policy model rather than as a separately trained network
- DPO trains a reward model identical to the one used in PPO-based RLHF
- DPO requires two separate reward models for stability
A: DPO still performs reward modeling, but implicitly, reparameterized directly from the policy model rather than as a separately trained network

Q: GRPO requires a learned critic/value network, just like PPO does.
- True
- False
A: False

Q: How does GRPO compute the advantage/baseline for a response, if not with a learned critic?
Feedback: Sample/expected answer: It samples multiple responses as a group for the same prompt and normalizes/compares each response's reward against the group's average (group-relative baseline), removing the need for a separate critic model.

Q: Your team has only unpaired binary feedback (each response individually labeled "desirable" or "undesirable," with no pairwise comparisons). Which method was designed specifically for this scenario?
- PPO
- DPO
- KTO
- Classic RLHF with a reward model
A: KTO

Q: Which method uses no reference model at all, making it the most memory-efficient option discussed this week but carrying a higher risk of catastrophic forgetting?
- GRPO
- DPO
- KTO
- ORPO
A: ORPO