🛠️ Lab 15 — Serve, Compare, and Hand Off Your Fine-Tuned Model (50 pts)

Goal: Serve the GGUF-quantized model you produced in Week 12 through at least two different local serving paths, compare their behavior and performance, and write a one-page handoff document recommending which artifact (unmerged adapter + base, or merged GGUF export) a downstream serving team should use — and why. All tools used must be free and runnable locally. Steps:

1. Retrieve your Week 12 artifacts. Locate your GGUF-quantized model file and, separately, your unmerged LoRA adapter (adapter weights + the base model you fine-tuned it on) from earlier weeks. If you no longer have the unmerged adapter, re-export it now — you need both forms for this lab.
2. Serve with Ollama (beginner-friendly path). Write a custom Modelfile that points FROM your GGUF file, and set a SYSTEM prompt reflecting your fine-tuning task. Run ollama create <your-model-name> -f Modelfile, then ollama run <your-model-name>. Send 3–5 prompts that specifically probe the fine-tuned behavior (not generic questions) and record the responses.
3. Serve the same model with llama.cpp server mode. Start llama.cpp's server (llama-server) with your GGUF file. Separately, if your hardware allows, start a second llama-server instance (or a second run) loading the unmerged base model plus the LoRA adapter directly (llama.cpp supports LoRA adapters at load time via its --lora flag). Send the same probe prompts to both and record responses, latency (wall-clock time per response is fine), and any differences in output quality.
4. (Optional stretch, if you have GPU access) Serve with vLLM. If you have access to a CUDA-capable GPU (local or a free-tier cloud notebook), load your base model in vLLM with --enable-lora and point it at your adapter directory. Send the same probe prompts and record latency/throughput. If you don't have GPU access, skip this step and note in your writeup that vLLM serving was evaluated conceptually only, citing the vLLM LoRA docs.
5. Compare your results. Build a small table comparing the paths you actually ran: setup difficulty, latency per response, whether fine-tuned behavior was clearly present in outputs, and whether the path supports hot-swapping additional adapters later.
6. Write your one-page handoff document. Address it to an imaginary "AIINFRA 200 serving team." State clearly: (a) which artifact you recommend they use going forward — the unmerged adapter + base, or the merged GGUF export — (b) your reasoning, grounded in this week's lecture (hot-swap needs vs. merge benefits, quantization-adapter compatibility risk, and what you personally observed), and (c) any risks or validation steps the serving team should not skip (e.g., confirming the adapter actually applied and did not silently fail).

Deliverables: Submit your Modelfile, your comparison table, your one-page handoff document (as a text entry or uploaded file), and a short log/screenshot of at least one probe-prompt exchange from each serving path you tested.