🛠️ Lab 10 — Benchmarking Base vs. Fine-Tuned Models with lm-evaluation-harness (50 pts)

Goal: Use EleutherAI's lm-evaluation-harness to quantitatively compare your Week 5 base model and fine-tuned model, and visually identify the point where your fine-tuning run began overfitting. Steps:

  1. Open a free Google Colab notebook (GPU runtime: Runtime → Change runtime type → T4 GPU) and install the harness: pip install lm-eval (or clone and install from the EleutherAI lm-evaluation-harness GitHub repo for the latest task definitions).
  2. Reload your Week 5 base model and your Week 5 fine-tuned model (from local storage, Google Drive, or your Hugging Face repo if you pushed one).
  3. Review the harness's Task Guide to confirm the exact task names you'll run, then evaluate the base model on a small, Colab-friendly slice of each: arc_challenge, gsm8k (if your fine-tune targeted math/reasoning), and one of hellaswag or winogrande. Use --limit to cap the number of examples per task so each run finishes in Colab's free-tier time limits.
  4. Run the identical harness command against your fine-tuned model, changing only the model path/checkpoint argument, so both runs are directly comparable.
  5. Build a simple results table (spreadsheet or Markdown) with one row per task and columns for base score, fine-tuned score, and delta (fine-tuned minus base).
  6. Flag any general-benchmark task (ARC-Challenge, HellaSwag, or WinoGrande) where the fine-tuned model dropped more than \~2–3 points versus the base model — this is your signal for possible catastrophic forgetting.
  7. Pull the training-loss and validation-loss values you logged during Week 5 fine-tuning (from your training script's logs or Trainer history) and plot both curves on one chart (train loss vs. eval loss over training steps/epochs) using matplotlib.
  8. Visually mark the point where validation loss stops decreasing and starts rising (or flattens while train loss keeps dropping) — this is your overfitting inflection point.
  9. If you saved intermediate checkpoints in Week 5, re-run the harness evaluation (step 3–4) on at least one earlier checkpoint near that inflection point and one later checkpoint, and add both to your results table to see whether benchmark performance diverges the way the loss curves predict.
  10. Write a short (200–300 word) summary interpreting your results: Did the fine-tune improve the target task? Did any general benchmark regress beyond the 2–3 point threshold? Where did overfitting begin, and does your checkpoint comparison confirm it? Would you ship this model as-is?

Deliverables: Submit your Colab notebook (.ipynb, shared link or exported file), your results table, your train-loss-vs-eval-loss plot (image or embedded in the notebook), and your written interpretation summary — via file upload or pasted text in the submission box.