🛠️ Lab 13 — Load-Test and Right-Size Your Capstone Endpoint (50 pts)

Goal: Write a Locust load test against your own deployed capstone inference endpoint, run it live, find the point where p99 latency starts to spike, and turn that into a right-sizing recommendation tied back to the SLO you set in Week 2. Free and local — no cloud spend required. Steps:

  1. Confirm your endpoint is running locally. Make sure your capstone API (FastAPI, or whatever you built in earlier weeks) is deployed and reachable on your local kind or minikube cluster, or running directly if that's where your capstone currently lives.
  2. Install Locust in your project's virtual environment (pip install locust). Confirm it starts with locust --version.
  3. Write locustfile.py. Follow the free "First Performance Test with Locust and FastAPI" walkthrough as your structural template. Define a User class with realistic wait_time (think-time between requests — do not set it to zero) and at least one @task that calls your actual capstone endpoint with realistic payloads (not the same static request every time).
  4. Add LLM-specific timing if your endpoint streams. If your capstone serves an LLM response as a stream, add custom timing in your task to capture time-to-first-token (record the timestamp when the first chunk arrives) and inter-token latency (gaps between subsequent chunks), not just total request time. Report these as custom Locust metrics or log them separately.
  5. Run a baseline test from the Locust web UI. Start Locust (locust -f locustfile.py), open the web UI, and run a modest number of simulated users representing your expected normal traffic. Record p95, p99, and RPS while the system is healthy.
  6. Ramp up to find the knee. Increase concurrent users in steps (e.g., 5, 10, 20, 40...) and re-run, watching RPS, latency percentiles, and failure rate live in the UI at each step. Identify the step where p99 stops being flat and starts spiking sharply — that's your practical capacity limit.
  7. Diagnose before you prescribe. Before deciding "add more replicas," check what's actually saturating: CPU/GPU utilization, memory, or request queuing. If utilization is low but latency still spikes, batching or concurrency configuration — not raw hardware — is likely your bottleneck.
  8. Right-size your deployment. Based on steps 6–7, adjust replica count or resource requests/limits on your kind/minikube deployment (or note the model-size change you'd make if the model itself is the bottleneck). Re-run the load test at your target traffic level to confirm the change moved your p99 in the right direction.
  9. Tie it back to your Week 2 SLO. State your original SLO, your measured p95/p99 at normal load, whether you met it, and your recommended right-sizing with justification.

Deliverables: Your locustfile.py, plus a load-test report (PDF or markdown, 1–2 pages) containing: a p95/p99 latency-vs-load curve (chart or table), the identified breaking point (knee) with supporting numbers, your diagnosis of the actual bottleneck, and a right-sizing recommendation explicitly tied back to your Week 2 SLO.