📖 Lecture — Compiling Models with TensorRT-LLM: From Checkpoint to Optimized Engine

So far in this course you have served models with vLLM and TGI — frameworks that load a Hugging Face checkpoint and start answering requests within minutes. This week introduces a fundamentally different philosophy: ahead-of-time (AOT) compilation. TensorRT-LLM, NVIDIA's inference library, does not "load and serve." Instead, it takes a model checkpoint and compiles it into a GPU-specific optimized binary called an engine, before any serving traffic ever arrives. It's the difference between an interpreted script and a compiled program: vLLM/TGI interpret the model graph on the fly for whatever GPU is present, while TensorRT-LLM produces a hardened, hardware-specific binary once and then serves from it repeatedly. The payoff is speed and efficiency at serving time; the cost is an upfront build step you must plan for.

The two-step compile pipeline

TensorRT-LLM's workflow always has two distinct stages:

  1. Quantize / convert the checkpoint. You take a standard Hugging Face checkpoint and convert it into TensorRT-LLM's internal checkpoint format, optionally applying quantization in the same step. For example:
python quantize.py --dtype float16 --qformat fp8
  1. Build the engine. The converted checkpoint is compiled into an optimized TensorRT engine using the trtllm-build CLI:
trtllm-build --checkpoint_dir ./tllm_checkpoint_1gpu_fp8

During the build step, TensorRT-LLM performs kernel fusion — merging multiple transformer operations (LayerNorm, matrix multiplications, bias-add, activation functions) into single, optimized CUDA kernels instead of launching each op separately. Fewer kernel launches means less overhead and better GPU utilization. Combined with a lower-precision numeric format, this is where the performance gains come from: NVIDIA reports that FP8 alone can roughly double throughput versus FP16/BF16 on supported hardware.

Choosing a quantization format

TensorRT-LLM supports several quantization schemes, and picking the right one depends on your GPU generation and your priority (accuracy vs. memory savings):

Format Precision Best for NVIDIA guidance
FP8 (per-tensor, block-scaling, rowwise) 8-bit float Hopper (H100) / Blackwell GPUs Recommended default — best accuracy/performance tradeoff
FP4 / NVFP4 4-bit float Latest Blackwell GPUs Newest option for maximum throughput where supported
INT8 SmoothQuant 8-bit integer Ada-generation GPUs (e.g., L40S) Recommended when FP8 hardware support isn't available
INT4 AWQ / GPTQ 4-bit integer Memory-constrained deployments Use when GPU memory, not compute, is the binding constraint

A good rule of thumb: start with FP8 if your GPU supports it, drop to INT8 SmoothQuant on Ada-class cards, and reach for INT4 AWQ/GPTQ only when you need to fit a model into a small memory footprint (as in this week's lab, where INT4 AWQ lets a 1.1B model compile and run comfortably on a free-tier GPU).

Correcting three common misconceptions

"A compiled engine should run on any NVIDIA GPU." It will not. A TensorRT-LLM engine is compiled for a specific GPU architecture, model architecture, and dtype combination. An engine built on an A100 will not run on an H100 or an L40S — you must rebuild for each target GPU. Plan your deployment pipeline around this: one engine per (model, GPU family, dtype) combination, not one engine for your whole fleet. "If the build fails, just re-run the same command." Re-running an identical failing command usually reproduces the identical failure, because stale intermediate artifacts from the previous attempt are often the actual cause. Before rebuilding, clear the build tree — rm -r build cpp/build — and then retry. This single habit resolves a large share of build errors that otherwise look mysterious. "Compiling is basically instant." It is not. A single-GPU build for a small model may take a few minutes, but multi-GPU tensor-parallel builds can take 10 to 90 minutes per (model, GPU, dtype) combination. That has real consequences: you cannot treat engine compilation like a lightweight pip install step in CI/CD. Budget build time explicitly, cache compiled engines wherever possible, and avoid triggering a full rebuild on every commit.

Why compile at all?

The tradeoff is straightforward: TensorRT-LLM trades serving-time flexibility for serving-time speed. Once compiled, an engine is highly optimized for exactly the workload it was built for — but it is inflexible to change (new GPU, new dtype, new parallelism strategy all require a rebuild). Later this week's discussion asks you to weigh this against vLLM/SGLang's more flexible, no-compile-step approach. For now, the mental model to keep is: quantize once, compile once, serve fast, many times.