🛠️ Lab 5 — Fine-Tune and Compare Dataset Formats with TRL SFTTrainer (50 pts)

Goal: Run a real supervised fine-tuning job end-to-end on a free GPU, and directly observe how dataset format (prompt-completion vs. conversational) affects the training process and the resulting model's output quality.

Steps

  1. Set up your environment. Open a new Google Colab notebook and set the runtime to a free T4 GPU (Runtime -\> Change runtime type -\> T4 GPU). Install TRL and its dependencies: pip install trl transformers datasets accelerate.
  2. Load the official TRL SFT workflow. Follow the structure of the official Hugging Face TRL SFT documentation (see Resources) to load a small base model — Qwen/Qwen2.5-0.5B-Instruct or google/gemma-3-270m — along with its tokenizer.
  3. Load a conversational dataset. Pick any small public instruction/chat dataset from the Hugging Face Hub (search for tags like conversational or instruction-tuning and choose one with a few hundred to a few thousand examples so training finishes in minutes). Confirm the format matches the {"messages": [{"role":..., "content":...}]} structure described in the lecture.
  4. Run Variant A: conversational format. Configure SFTTrainer with this dataset, leaving completion_only_loss at its default (True). Train for a small number of steps or one epoch. Record the loss curve (screenshot or copy the printed loss values across steps).
  5. Build Variant B: prompt-completion format. Convert a copy of the same dataset (or a comparable one) into plain {"prompt": ..., "completion": ...} fields rather than message lists. Train a second SFTTrainer run with identical hyperparameters (same model, same number of steps/epoch, same learning rate).
  6. Generate before/after completions. Before fine-tuning, prompt the base model with 3 sample questions/instructions from your domain and record its raw outputs. After each of Variant A and Variant B finish training, prompt the fine-tuned models with the same 3 questions and record their outputs.
  7. Compare and analyze. Build a short table comparing: loss curve shape for A vs. B, and output quality for base vs. A vs. B on your 3 test prompts. Note any differences in fluency, instruction-following, or formatting.
  8. Export your notebook. Save your completed Colab notebook (File -\> Download -\> Download .ipynb), including all cell outputs (loss logs and generated completions).

Deliverables: Submit your exported .ipynb notebook (with outputs visible) plus a short written summary (200–400 words) comparing the two dataset-format variants — their loss curves and their generated completions — and stating which format you'd recommend for a real project and why.