Title: Lecture 9 — From RLHF to DPO to GRPO: Preference Tuning Without the RL Headache
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 09 — Preference Tuning: DPO in Practice (with a Look at GRPO)

📖 Lecture — From RLHF to DPO to GRPO: Preference Tuning Without the RL Headache

By now you know how to fine-tune a base model on demonstrations (AIINFRA 200/202 earlier weeks) so it follows instructions. But instruction tuning alone often produces a model that's technically correct yet stylistically flat, unhelpful in the ways that matter to actual users, or subtly unsafe. The fix the field converged on is preference tuning: instead of teaching the model "here is the right answer," you teach it "here is the answer humans (or a reward signal) prefer over another." This week is about the two dominant families of preference tuning you'll actually use in practice: DPO and GRPO.

The RLHF baseline, and why it's painful

Classic Reinforcement Learning from Human Feedback (RLHF) is a two-stage process. First you train a reward model on human preference judgments (given two responses to the same prompt, which one do people like better?). Then you use that reward model as the signal for PPO (Proximal Policy Optimization) to actually update your language model's weights through reinforcement learning. This works, but it is operationally heavy: PPO-based RLHF requires four models in memory simultaneously — the policy model being trained, a frozen reference model (to keep it from drifting too far), the reward model, and a value function/critic that estimates expected future reward. Four models means four times the GPU memory footprint, plus RL-specific instabilities: sensitive hyperparameters, reward hacking, and sampling variance that can make a training run diverge for reasons that are hard to diagnose.

DPO: reparameterizing the problem so RL disappears

Direct Preference Optimization (DPO) asks a clever question: what if we could mathematically show that the optimal policy under the RLHF objective has a closed-form relationship to the reward function? It turns out you can substitute that relationship back into the preference-learning objective and get a loss you can optimize directly with supervised-style gradient descent on pairs of (chosen, rejected) responses — no reward model training, no PPO rollouts, no RL sampling instability. The practical result: DPO needs only two models in memory (the policy being trained, and a frozen reference copy of the starting model), and the trainer implementation is famously compact — on the order of 20 lines of core loss logic. That is why libraries like TRL's DPOTrainer have made preference tuning accessible to teams without dedicated RL infrastructure.

The frozen reference model still matters

Because DPO has no explicit RL loop, it's tempting to think it has thrown away RLHF's safety rails too. It hasn't. The frozen reference model in DPO plays the same role the KL-penalty term played in PPO-based RLHF: it provides a soft KL-divergence constraint that keeps the policy from wandering too far from its original, well-behaved distribution while it optimizes for preferences. If you mishandle the reference model — for example, by letting it drift, using the wrong checkpoint, or setting the beta (KL-strength) hyperparameter poorly — training can destabilize: the model can overfit to the preference pairs, degenerate into repetitive or extreme outputs, or lose general capability. Treat the reference model as a first-class hyperparameter choice, not an implementation detail.

GRPO: dropping the critic for group-relative rewards

Group Relative Policy Optimization (GRPO) takes a different path to the same goal of simplifying RLHF, and it has become the preference-tuning method of choice for reasoning-heavy tasks (as seen in DeepSeek-R1-style training). Where PPO needs a learned value/critic network to estimate a baseline for computing advantage, GRPO eliminates the critic entirely. Instead, for each prompt, GRPO samples a group of multiple responses, scores each with a reward function, and computes each response's advantage relative to the average reward within that group. No critic means one fewer model in memory versus PPO, and the group-relative baseline turns out to work especially well when rewards come from verifiable signals (like "did the math answer match?") rather than from noisy pairwise human judgment.

Choosing the right tool: a comparison

Method | Models in memory | Feedback format | Needs reward model? | Needs reference model? | Best suited for
PPO-based RLHF | 4 (policy, reference, reward, value/critic) | Preference pairs → trained reward model | Yes (explicit, trained) | Yes | General-purpose RLHF, when you have infra to support it
DPO | 2 (policy, reference) | Preference pairs (chosen vs. rejected) | Implicit only | Yes | Style/helpfulness tuning with a fixed preference dataset
GRPO | 2 (policy, reference), no critic | Group of sampled responses + reward function | Optional (rule-based or model-based) | Yes | Reasoning tasks with verifiable/scorable rewards
KTO | 2 (policy, reference) | Unpaired binary desirable/undesirable labels | Implicit only | Yes | When you only have thumbs-up/down data, not pairs
ORPO | 1 (policy only) | Preference pairs | Implicit only | No | Maximum memory efficiency; higher forgetting risk

Two misconceptions worth correcting directly

"DPO eliminates the reward model entirely." Not quite. DPO eliminates the separate, explicitly trained reward model and the RL loop that uses it — but the math underneath DPO reparameterizes reward as a function of the policy and reference model's log-probabilities. In other words, DPO still performs reward modeling; it just does it implicitly, folded directly into the policy update. The paper's title says it plainly: "Your Language Model is Secretly a Reward Model." "GRPO still needs a learned critic like PPO." It does not. This is GRPO's entire point of departure from PPO: instead of training a value/critic network to estimate a baseline, GRPO samples multiple responses per prompt and uses the group's own reward statistics (mean, and often standard deviation) as the baseline for computing each response's relative advantage. This removes an entire model from memory and removes a whole class of critic-training instability, which is part of why GRPO scales well on reasoning workloads with limited compute.

Rounding out the family: KTO and ORPO

Not every team has clean preference pairs. KTO (Kahneman-Tversky Optimization) was designed for exactly that gap: it works with unpaired binary feedback — just "this output was good" or "this output was bad" labels, no need to have two responses to the same prompt ranked against each other. ORPO (Odds Ratio Preference Optimization) goes even further on the memory-efficiency axis by dropping the reference model altogether, folding a preference penalty directly into the supervised fine-tuning loss. That makes ORPO the leanest option in terms of models-in-memory, but because it has no reference model anchoring it to prior behavior, it carries a higher risk of catastrophic forgetting of the base model's general capabilities. The throughline for this week: every one of these methods is solving the same core problem (align a model to preferences without the operational burden of full RLHF), but they differ in what data they assume you have and how much memory/stability they trade away to get there. Your lab this week will let you feel these tradeoffs directly, by running both a DPO pass and a GRPO pass on the same free-tier hardware.