Lecture 7 — NVIDIA Triton Inference Server and Model Repositories

📖 Lecture — NVIDIA Triton Inference Server and Model Repositories

So far in this course you've served models with single-purpose engines. This week we introduce a different philosophy: NVIDIA Triton Inference Server, an open-source inference server (currently at release v2.70.0, shipped in the NGC container tagged 26.06) built to be the one server that serves everything. Instead of standing up a separate serving process for each framework, Triton lets you point one server instance at a directory of models — PyTorch, TensorFlow, ONNX, TensorRT, even arbitrary Python code — and it serves all of them simultaneously over a standard HTTP or gRPC protocol. For a platform team supporting multiple data science groups with different framework preferences, this consolidation is the whole appeal: one set of ports, one set of metrics, one deployment to operate. The model repository: Triton's core abstraction Everything Triton serves lives in a model repository — a directory tree that Triton scans at startup (and optionally polls for changes). Each model gets its own subdirectory, and inside that subdirectory Triton expects a very specific layout:

Path	Purpose
`<model-repository>/<model-name>/`	Top-level directory for one model
`<model-repository>/<model-name>/config.pbtxt`	ModelConfig protobuf: name, backend, max batch size, input/output tensor shapes and types
`<model-repository>/<model-name>/<version>/`	A numeric sub-directory (e.g. `1/`, `2/`) holding the actual model file(s) for that version

This is the single biggest source of early confusion, and the brief that shaped this course flags it explicitly, so let's correct it head-on: Triton will refuse to load an otherwise perfectly good model if it does not find at least one numeric version sub-directory. It is not enough to drop model.onnx directly into the model's top-level folder — it must sit inside a folder literally named 1 (or 2, 3, and so on for subsequent versions). This versioning structure is how Triton supports side-by-side model versions and controlled rollouts, but until you've hit the error once, it feels like an arbitrary hoop to jump through. Budget time in the lab this week to get this right. A second misconception worth correcting up front: many beginners assume every model needs a hand-written config.pbtxt. That's not strictly true — for several supported backends and formats (notably TensorFlow SavedModel and ONNX, when the tensor shapes can be inferred from the model file itself), Triton can auto-generate a minimal configuration at load time. That said, in real deployments you will almost always want to hand-edit config.pbtxt anyway, because that's where you control dynamic batching windows, instance groups (how many copies of the model run, and on which GPUs), and version policy — none of which auto-generation will infer for you. Backends for LLM serving Triton isn't just for traditional CV/NLP models — it has first-class backends for large language model serving. The vLLM backend, available since the 23.10 container, is architecturally elegant: it takes every incoming request and hands it directly to vLLM's AsyncEngine, letting vLLM itself manage inflight (continuous) batching and PagedAttention memory management. Triton, in this configuration, is essentially a protocol front-end around vLLM's own scheduler. The separate TensorRT-LLM backend takes a different approach: it uses a purpose-built C++ component called inflight_batcher_llm to serve models that have already been compiled into optimized TensorRT engines. The trade-off is familiar from earlier weeks — vLLM backend gives you flexibility and fast iteration with standard Hugging Face checkpoints, while TensorRT-LLM gives you maximum throughput once you're willing to invest in the engine-compilation step. Chaining models: ensembles and BLS Real production pipelines rarely involve just one model — you typically need preprocessing, inference, and postprocessing steps, and sometimes several models in sequence. Triton gives you two mechanisms for this, and picking the right one matters:

Ensembles chain models declaratively, entirely inside Triton configuration. You define a pipeline (e.g., preprocess -> resnet50 -> postprocess) by mapping each step's output tensors to the next step's input tensors. Triton handles execution and data movement internally — the client sends one request and gets one response, with zero client-side orchestration code.
Business Logic Scripting (BLS) exists for the cases ensembles can't handle: pipelines that need loops, conditional branching, or dynamic decisions about which model to call next. With BLS, a Python-backend model acts as an orchestrator, programmatically calling other models on the same Triton instance and combining their outputs however your logic requires.

A good rule of thumb: reach for an ensemble first, because it's simpler and fully declarative. Reach for BLS only when your pipeline genuinely needs control flow that a static tensor-mapping graph can't express. When Perf Analyzer fails, don't panic One more misconception to retire: when Triton's Perf Analyzer tool throws an error, the instinctive reaction is to assume the underlying model is broken. In practice, the most common root cause is far more mundane — a mismatch between the input/output shapes declared in config.pbtxt and the shapes the model actually expects. Before you go hunting for a deeper bug, always sanity-check your config against the model's real signature first. It will save you a lot of debugging time this week and in your career afterward.