🛠️ Lab 13 — Benchmarking Cost per 1,000 Requests with vLLM on a Free GPU (50 pts)

Goal: Use a free T4 GPU and the open-source vLLM serving engine to measure real throughput and latency for a small language model, then convert those measurements into cost-per-1,000-requests and cost-per-million-tokens estimates using published GPU hourly rates — and empirically find the batching "knee of the curve." Steps:

1. Open a free Google Colab notebook (or a free Kaggle notebook) and select a T4 GPU runtime.
2. Install vLLM in the notebook (pip install vllm) and pick a small open model that fits comfortably on a T4 (for example, a 1–3B parameter instruction-tuned model). Confirm the model loads and serves a single test prompt successfully before moving on.
3. Launch vLLM's OpenAI-compatible server in the background within the notebook, pointed at your chosen model.
4. Run vLLM's built-in benchmark_serving script against your running server to measure baseline throughput (tokens/sec) and latency (time-to-first-token and per-token latency) at a low concurrency level (e.g., 1 concurrent request).
5. Repeat the benchmark_serving run at increasing concurrency/batch-size levels (e.g., 1, 4, 8, 16, 32 concurrent requests), recording throughput and per-request latency at each level.
6. Chart your results: throughput (tokens/sec) on one axis and per-request latency (ms) on the other, both against concurrency. Identify the "knee of the curve" — the point where additional concurrency stops meaningfully improving throughput but continues to hurt latency.
7. Using the reference formula cost per million tokens = (cluster \$/hr) / (tokens/sec × 3600 / 1,000,000), plug in one published GPU hourly rate from the lecture's pricing table (your choice — cite which one) at your lowest and your knee-of-the-curve concurrency levels. Show both calculations.
8. Convert your knee-of-the-curve cost-per-million-tokens figure into a cost-per-1,000-requests estimate, assuming a realistic average prompt+completion size of your choosing (state your assumption, e.g., "500 tokens per request").
9. Write a short (150–250 word) summary explaining: (a) how much cost per token changed between your lowest and highest concurrency runs, (b) where the knee of the curve landed and why, and (c) whether your chosen GPU/model pairing looked over-tiered or right-sized for the load you tested.

Deliverables: Submit your notebook (or a link/export of it) showing the benchmark_serving output at each concurrency level, your throughput-vs-latency chart, your two cost calculations (lowest concurrency and knee-of-the-curve), and your written summary — via online upload or pasted directly into the text entry box.