Title: Discussion 10 — Ship It or Iterate? Weighing Trade-offs in Fine-Tune Evaluation
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 10 — Evaluating Fine-Tuned Models and Detecting Overfitting

💬 Discussion 10 — Ship It or Iterate? Weighing Trade-offs in Fine-Tune Evaluation (20 pts)

Prompt: Your fine-tuned model scored +5 points on your custom task but −3 points on MMLU compared to the base model. Is that a good outcome? Using this week's lecture (especially the idea of regression testing and the "ship only if regression holds steady AND improvement is statistically meaningful" rule), explain how you would decide whether to ship this model or continue iterating. Be specific: what additional evidence would you want before deciding (e.g., statistical significance, which benchmarks, checkpoint comparisons), and what would make you say "ship it" versus "go back to training"? Feel free to connect this to what you saw in your own Lab 10 results. Post your initial response (150+ words), then respond to at least two classmates — look for a post that reached a different conclusion than yours and explain, with reference to the lecture's evaluation structure, why you'd weigh the trade-off differently (or the same).