🛠️ Lab 4 — Measuring Prefix Caching and Concurrency Gains in vLLM (50 pts)

Goal: Using the same free-tier vLLM deployment from Week 3 (Google Colab T4 GPU, or a local GPU if you have one), empirically observe and quantify how prefix caching and continuous batching affect throughput and latency — turning this week's concepts into numbers you measured yourself. Steps:

1. Relaunch your Week 3 vLLM server (Colab T4 notebook or local install). Confirm it responds to a single test request before moving on.
2. Baseline — single request. Send one request to the server and record time-to-first-token and total generation time for a fixed prompt and fixed output length (e.g., 200 tokens). This is your single-request baseline.
3. Concurrency test — without prefix caching. Launch the server without --enable-prefix-caching. Using a simple Python script (asyncio + httpx, or a basic loop with concurrent.futures) or a load tool like vegeta or hey (all free/local), fire off concurrent requests at increasing levels — try 1, 4, 8, and 16 concurrent requests — using prompts that share a common long prefix (e.g., the same system prompt or few-shot examples, with only the final question varying). Record total throughput (tokens/sec across all requests) and average per-request latency at each concurrency level.
4. Concurrency test — with prefix caching. Restart the server with --enable-prefix-caching enabled. Repeat the exact same concurrency sweep (1, 4, 8, 16) with the same shared-prefix prompts. Record the same metrics.
5. Vary batch size / concurrency and graph it. Using a spreadsheet or a short Python script (matplotlib is free and local), plot two graphs: (a) throughput vs. concurrency level, with and without prefix caching as two lines; (b) average latency vs. concurrency level, with and without prefix caching as two lines.
6. Interpret continuous batching in action. Compare your single-request baseline (Step 2) throughput per request to the per-request throughput you observed at concurrency 8 or 16. Write 3-5 sentences explaining what you observed in terms of continuous batching: is the GPU clearly serving multiple requests without waiting for each to finish? Does total system throughput scale roughly linearly, sub-linearly, or does it plateau — and why would you expect that given what you learned about block-based memory limits this week?

Deliverables: Submit (a) your benchmarking script or notebook, (b) your raw results table (concurrency level × throughput × latency, with/without prefix caching), (c) your two graphs (throughput and latency vs. concurrency), and (d) your written interpretation from Step 6 (150-250 words) connecting your data back to PagedAttention and continuous batching concepts from the lecture.