Title: Discussion 8 — Whose Benchmark Do You Trust?
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 08 — Serving Engine Benchmarking Lab

💬 Discussion 8 — Whose Benchmark Do You Trust? (20 pts)

Prompt: Benchmarking tools and published papers frequently disagree about which serving engine is "best" — one comparison shows TensorRT-LLM ahead of vLLM at every concurrency level, another shows TGI dramatically ahead on long-context workloads, and different tools even disagree on how to combine TTFT and TPOT into a single number. In a post of at least 200 words, answer:

- Which methodology choices (prompt length/distribution, concurrency level, metric definitions, hardware, dataset realism) do you think most significantly change a benchmark's reported "winner," and why?
- How should a practitioner responsibly interpret two conflicting benchmark claims about the same engines, rather than just picking whichever result is more convenient?
- Bring in at least one specific point from this week's lecture or resources (e.g., the TTFT/TPOT methodology disagreement, or the ShareGPT vs. synthetic-prompt discussion) to ground your answer.

Respond to two peers: For each peer, either (a) push back with a methodology detail they didn't consider, or (b) describe a concrete scenario at your own organization where their conclusion would or wouldn't hold, and explain why.