🛠️ Lab 9 — DPO and GRPO Head-to-Head on a Free Colab T4 (50 pts)

Goal: Get hands-on with both major preference-tuning methods using only free tools. You'll run TRL's DPOTrainer on a preference-pairs dataset to observe behavioral shifts before/after tuning, then run a GRPO exercise with Unsloth to reproduce a mini reasoning-model training run — so you can compare the two reward signals directly instead of just reading about them. Steps:

  1. Open a free Google Colab notebook and set the runtime to a T4 GPU (Runtime → Change runtime type → T4 GPU).
  2. Install TRL and dependencies (pip install trl transformers datasets accelerate peft) and load a small base model you can afford on a T4 (a \~1–3B parameter instruct model works well for this exercise).
  3. Load a 1,000-sample subset of either Intel/orca_dpo_pairs or Anthropic/hh-rlhf from the Hugging Face Hub. Do not load the full HH-RLHF (\~160k examples) — it is too slow for the free tier. Use dataset slicing (e.g., dataset.select(range(1000))) to cap it.
  4. Before training, generate responses from the base model on 5 held-out prompts and save them — this is your "before" snapshot.
  5. Configure and run TRL's DPOTrainer, pointing it at your policy model and a frozen copy of the same model as the reference. Train for a small number of steps appropriate to free-tier time limits (document your exact hyperparameters: learning rate, beta, batch size, steps).
  6. After training completes, generate responses on the same 5 prompts with the newly DPO-tuned model — this is your "after" snapshot.
  7. Write a short before/after comparison: what changed in tone, helpfulness, length, or refusal behavior? Note anything that looks like over-optimization (e.g., repetitive phrasing, obvious reward-hacking artifacts).
  8. In a second Colab T4 notebook, follow the Hugging Face LLM Course GRPO exercise using Unsloth (Unsloth's optimizations let you handle models up to roughly 16B parameters on a single T4). Reproduce the mini reasoning-model training run as described in the exercise.
  9. Record the reward function used in your GRPO run and briefly describe how group-relative advantage was computed for your sampled responses.
  10. Write a short comparison section: how did GRPO's group-relative reward signal feel different to configure and interpret compared to DPO's pairwise-preference approach? Which would you reach for if your task were "make responses sound less robotic" vs. "make math answers more often correct"?

Deliverables: Submit both notebooks (as .ipynb files or shared Colab links) plus a short written report (1–2 pages, online text entry or upload) containing your before/after DPO generations, your documented hyperparameters, your GRPO reward function and setup notes, and your comparison analysis.