🛠️ Lab 15 — GPU Observability Stack and Cost-Per-1000-Tokens Model (50 pts)

Goal: Stand up a free, fully local GPU observability stack (DCGM Exporter + Prometheus + Grafana) in front of a running vLLM server, visualize real GPU telemetry with NVIDIA's official Grafana dashboard, correlate utilization spikes with load-testing traffic from Week 8, and then build a cost-per-1000-tokens spreadsheet model that quantifies the dollar impact of utilization and quantization choices. Steps:

1. Stand up the stack with Docker Compose. Using only free, local tools, write a docker-compose.yml that runs three services: dcgm-exporter (NVIDIA's official image), prometheus, and grafana. Configure Prometheus to scrape the DCGM exporter's /metrics endpoint on its default interval. Confirm all three containers are healthy with docker compose ps.
2. Point Prometheus at your vLLM server too. Add a scrape target for your vLLM server's /metrics endpoint (vLLM exposes Prometheus-format metrics natively). Verify in the Prometheus web UI (Status → Targets) that both the DCGM exporter and vLLM targets show as "UP."
3. Import the official Grafana dashboard. In Grafana, add Prometheus as a data source, then import NVIDIA's pre-built "NVIDIA DCGM Exporter Dashboard" using dashboard ID 12239. Confirm panels for GPU utilization, memory-copy (bandwidth) utilization, and temperature are populating with live data.
4. Generate load and correlate. Re-run (or replay) the load-testing script/traffic pattern you built in your Week 8 benchmarking lab against your vLLM server. Watch the Grafana dashboard in real time (or review it immediately after) and screenshot the utilization, memory-copy, and temperature panels during the traffic spike. In your written submission, describe what you observed: did GPU-Util rise in step with your request rate? Did memory-copy utilization tell a different story than GPU-Util? Did vllm:num_requests_waiting climb, and if so, when?
5. Pull the numbers you need for costing. From your Grafana/Prometheus data (or nvidia-smi/DCGM logs if you prefer a simpler capture), record the average GPU utilization percentage during your load test, and note your model's approximate output tokens/sec throughput at that utilization level.
6. Build the cost-per-1000-tokens spreadsheet. Using a free spreadsheet tool (Google Sheets or LibreOffice Calc), build a model with at least these two scenarios side by side:
• Underutilized: \~20% GPU utilization, no batching, using published on-demand GPU hourly pricing for your chosen GPU (e.g., H100 at \~\$2.69/hr).
• Right-sized: \~80% GPU utilization achieved through batching (and, optionally, a lower-precision format like 4-bit AWQ on cheaper hardware), using the corresponding hourly price.

For each scenario, calculate: GPU-hours needed to serve 1,000,000 tokens, cost of those GPU-hours, and the resulting cost per 1,000 tokens. Show your formulas, not just final numbers.

1. Quantify the dollar impact. Add a summary row/section that states, in plain language, the percentage cost reduction and estimated annual dollar savings if this workload ran at the right-sized configuration instead of the underutilized one (you may assume a steady request volume of your choosing — state your assumption clearly).

Deliverables: A short written report (1–2 pages, submitted as text entry or upload) containing your docker-compose.yml, Grafana dashboard screenshots from the load-test correlation step, your observations about utilization vs. queue depth, and your cost-per-1000-tokens spreadsheet (uploaded as a file or linked) with the underutilized-vs-right-sized comparison and dollar-impact summary.