Title: Discussion 6 — Compile-Ahead vs. Serve-and-Adapt
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 06 — TensorRT-LLM Compilation and Optimization

💬 Discussion 6 — Compile-Ahead vs. Serve-and-Adapt (20 pts)

TensorRT-LLM takes an ahead-of-time (AOT) compilation approach: you quantize and compile a GPU-specific engine before serving begins. vLLM and SGLang, by contrast, load a model checkpoint and serve it just-in-time (JIT), adapting to whatever GPU is available at runtime with no separate compile step. In your initial post, address:

- What are the flexibility tradeoffs? Which approach adapts more gracefully to a fleet with mixed GPU types (e.g., a mix of A100s and H100s)?
- What are the performance tradeoffs? Where does TensorRT-LLM's upfront compilation investment pay off, and where might it not be worth it?
- What are the operational-complexity tradeoffs? Consider build times, CI/CD pipeline design, and how each approach affects how quickly your team can iterate or roll back a model change.

Post your initial response (200+ words), then respond to at least two classmates — ideally engaging with someone who reached a different conclusion than you did about which approach they'd choose for a production deployment.