📖 Lecture — From TGI to SGLang: The New Serving Engine Landscape

For years, Hugging Face's Text Generation Inference (TGI) was the default answer to "how do I serve an LLM in production?" It popularized techniques that are now table stakes across the entire industry: continuous batching, which packs newly arrived requests into an already-running batch instead of waiting idle for the slowest sequence to finish; tensor parallelism, which splits a model's weight matrices across multiple GPUs so a single request can be served faster than one GPU could manage alone; and FlashAttention, a fused, memory-efficient attention kernel that dramatically cuts the memory bandwidth cost of the attention operation. If you have used any modern serving engine, you have been standing on TGI's shoulders whether you knew it or not. That said, an important status change happened in late 2025: TGI entered maintenance mode and is now read-only. Hugging Face itself now recommends vLLM, SGLang, llama.cpp, or MLX for new deployments, depending on your hardware and use case. This is not a knock on TGI's engineering — it is a signal about how fast this ecosystem moves. TGI's core ideas were good enough that competing projects absorbed them and then kept innovating past the point where TGI's maintainers could keep pace. Studying TGI's design is still worthwhile, not because you'll deploy it new, but because it is the shared ancestor of the techniques every modern engine uses. That brings us to SGLang, one of the two engines (alongside vLLM, which you covered in a previous course) most commonly recommended today. SGLang's headline innovation is RadixAttention. Instead of treating each incoming request's KV-cache as an isolated block of memory, RadixAttention maintains a radix tree that indexes token prefixes across all requests being served. When a new request arrives, SGLang walks the tree to find the longest prefix it already has cached from a previous request, and reuses that cached KV state instead of recomputing it. On workloads where prompts share 60% or more of their tokens as a common prefix — think a RAG system that prepends the same retrieved documents to many user queries, or a multi-turn chatbot where each turn re-sends the entire conversation history — this delivers dramatic reductions in time-to-first-token (TTFT), because the engine skips recomputing attention over tokens it has already seen. It's tempting to think of RadixAttention as "just a smarter cache," but it's more foundational than that. Combined with zero-overhead CPU scheduling (the scheduler's bookkeeping is overlapped with GPU compute instead of stalling it), continuous batching, paged attention for efficient KV-cache memory management, and support for both tensor and pipeline parallelism, SGLang is a full-featured serving engine and a direct competitor to vLLM — not merely a caching layer bolted onto someone else's engine. The table below summarizes how the three techniques stack up:

To get hands-on, you'll install SGLang with pip install "sglang[all]" and launch a server with a command like:

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 30000

Now let's address three misconceptions students commonly bring into this material. Misconception 1: "TGI is still the cutting-edge choice because it was historically Hugging Face's flagship engine." This was true for years, but it is no longer accurate. As of late 2025, TGI is in maintenance mode and read-only. For any new production deployment, Hugging Face itself directs you to vLLM or SGLang (or llama.cpp/MLX for edge and Apple Silicon use cases). Use TGI as a historical reference for understanding why continuous batching and tensor parallelism became standard — not as a recommendation for what to deploy today. Misconception 2: "SGLang is just vLLM with a different name," since both use paged KV-cache management. The two engines do share paged attention as a memory-management strategy, but that's where the similarity ends. SGLang's distinguishing feature is RadixAttention: a radix tree that indexes KV-cache entries at the token level so that any two requests sharing a prefix — even ones submitted independently, not just turns in the same conversation — automatically reuse that shared computation. This is especially valuable for RAG pipelines (shared retrieved context) and multi-turn chat (shared conversation history), where the prefix-overlap benefit compounds with every additional request. Misconception 3: "All quantization methods are interchangeable — they just save memory." Not so. Quantization method choice carries real throughput tradeoffs, not just memory tradeoffs. For example, bitsandbytes quantizes weights automatically at load time, which is convenient, but it runs inference slower than GPT-Q or FP16 in many configurations because of how its dequantization is implemented at inference time. GPT-Q and AWQ, by contrast, are calibrated ahead of time and are generally optimized for faster inference at similar memory savings. When you're choosing a quantization method for a production deployment, always benchmark throughput and latency — don't assume "quantized" means "quantized the same way." Understanding why TGI was retired while vLLM and SGLang thrived is itself a lesson in open-source sustainability: a project's early lead doesn't guarantee its long-term position, and in a fast-moving field, the engines that keep absorbing new research (like RadixAttention) and maintain active contributor communities are the ones that last.