🛠️ Lab 5 — TGI vs. SGLang Head-to-Head Benchmark (50 pts)

Goal: Stand up a legacy TGI server alongside a modern SGLang server on free/local GPU resources, send each engine identical prompts, and observe firsthand how RadixAttention's prefix-caching advantage shows up in real throughput and latency numbers. Steps:

  1. Open a free GPU runtime (Google Colab or Kaggle, T4 GPU) or use a local GPU machine if you have one available.
  2. Install Rust (required to compile TGI from source) and compile Text Generation Inference, following the Towards Data Science walkthrough linked in this week's Resources.
  3. Launch the TGI server with an AWQ-quantized model so you can serve a reasonably sized model on limited GPU memory:
text-generation-launcher --model-id TheBloke/Llama-2-13B-chat-AWQ --quantize awq --port 5000
  1. In a separate terminal/session, install SGLang:
pip install "sglang[all]"
  1. Launch the SGLang server with a small instruct model so it runs comfortably alongside TGI on the same free GPU:
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --port 30000
  1. Write a short Python or shell script that sends the same set of 8–10 prompts to both servers (port 5000 for TGI, port 30000 for SGLang). Include at least 3 prompts that share a long common prefix (e.g., the same "system instructions + retrieved context" block, with only the final question changing) to simulate a RAG workload.
  2. Record wall-clock latency and (if available) tokens/sec for each request against each server. A simple table or spreadsheet is fine.
  3. Re-run the shared-prefix prompts a second time against each server and compare whether SGLang's response times drop on the repeat run (this is where RadixAttention's prefix reuse should become visible) versus TGI, which has no equivalent cross-request cache reuse mechanism.
  4. Write up a short (300–500 word) summary explaining what you observed, why you think SGLang behaved the way it did on the repeated-prefix prompts, and one production scenario where you would choose SGLang over TGI (or vice versa) based on this evidence.

Deliverables: Submit your benchmark script or notebook, your recorded latency/throughput table, and your written summary (as an upload or pasted into the text entry box).