Title: Lecture 9 — Serving, Adapting, or Augmenting: Choosing Your Model Layer
Course: AIINFRA 303: Capstone Project > Section: Week 09 — The Model Layer: Serving, Adaptation, or RAG Integration

📖 Lecture — Serving, Adapting, or Augmenting: Choosing Your Model Layer

Every capstone project eventually has to answer one question: how does your application actually talk to a model? Across AIINFRA 100–302 you built the toolchain — containers, cloud deployment, CI/CD, inference serving basics, fine-tuning, agents, RAG, security, and FinOps. This week you integrate those pieces into the "model layer" of your own capstone: the component that decides how a model is served, how it is adapted to your task, or how it is grounded in your own data. There are three real options — serving an open model efficiently, adapting a model with LoRA, or grounding a model with RAG — and picking the right one (and knowing why) is itself a core infrastructure skill. Serving open models with vLLM. If your capstone needs a model you fully control, running on your own infrastructure, vLLM is the production standard for open-weight serving. Its core innovation is PagedAttention: rather than allocating one large contiguous block of GPU memory per sequence for the KV cache (the running memory of attention keys/values), vLLM manages the KV cache in small, non-contiguous "pages," much like virtual memory paging in an operating system. This nearly eliminates memory fragmentation and lets vLLM pack far more concurrent requests onto the same GPU, which is what drives its throughput advantage over naive Hugging Face generate() loops. Getting an OpenAI-compatible endpoint running is a single command: vllm serve <model>. Because the API shape matches OpenAI's chat completions format, any code you already wrote against OpenAI's client library (or LangChain's ChatOpenAI wrapper pointed at a custom base_url) works against your self-hosted model with minimal changes. Adapting models with LoRA. If the base model's knowledge is fine but its behavior, tone, or output format is not, full fine-tuning is rarely the right tool — it is expensive in GPU-hours, slow to iterate on, and risks catastrophic forgetting. LoRA (Low-Rank Adaptation), via Hugging Face's PEFT library, freezes every weight in the base model and instead learns a pair of small low-rank matrices injected alongside select weight matrices (typically the attention projections). Because the rank is small (often 8–64) relative to the full weight matrix dimensions, LoRA typically cuts trainable parameters by roughly two orders of magnitude versus full fine-tuning — meaning you can adapt a model on a single consumer or free-tier GPU in an hour rather than needing a multi-GPU cluster for days. The adapter is also a small, portable file you can swap in and out of the frozen base model, which is why LoRA has become the default adaptation technique in production ML infrastructure. Grounding models with RAG. If your capstone's core problem is that the model doesn't know your data — internal documents, a changing knowledge base, domain-specific facts — RAG (Retrieval-Augmented Generation) is almost always the right first move, and it doesn't touch model weights at all. RAG is a two-step pipeline:

Step | What happens | Key components
1. Indexing (offline) | Documents are split into chunks, each chunk is embedded into a vector, and vectors are stored in a vector database | Document loader, text splitter, embedding model, vector store (e.g., ChromaDB)
2. Retrieval + generation (online) | The incoming query is embedded, similarity-matched against stored vectors to retrieve the most relevant chunks, and those chunks are concatenated with the query into a prompt sent to the LLM | Embedding model, similarity search, prompt template, LLM

The single biggest infrastructure benefit of RAG is that your knowledge base can be updated completely independently of the model's parameters — add, edit, or remove documents and the next query immediately reflects the change, with zero retraining and zero downtime. LangChain (and its stateful-agent counterpart LangGraph) reached a stable v1.0 release in October 2025, and the official LangChain RAG tutorial walks through the exact pipeline above: document loading, chunking, embedding, and retrieval-augmented generation. Correcting three common misconceptions. First, fine-tuning is the most over-applied option in the field, not the default first resort. It is the most expensive and slowest option to iterate on, so the right order of operations is: start with prompt engineering, reach for RAG when the problem is external or changing knowledge, and reserve fine-tuning for behavior you genuinely cannot get any other way (a specific tone, a rigid output schema, a skill the base model lacks entirely). Second, when a RAG system produces a wrong or hallucinated answer, the instinct is to blame the generator — but retrieval quality sets the ceiling on generation quality, not the other way around. Most RAG failures start upstream, in document parsing, chunking, or indexing, so always debug the retrieval pipeline (what did it actually retrieve?) before you touch prompts or swap models. Third, smaller chunks are not automatically better. Tiny chunks fragment context, forcing the model to reason over disconnected fragments, and they can actually inflate cost by requiring more chunks per query to cover the same information. Chunk on semantic boundaries — paragraphs, sections, logical units — with a slight overlap between chunks, rather than slicing by a fixed byte or token count. By the end of this week, you should be able to look at your own capstone's requirements and confidently say which of these three tools is the right one to reach for — and defend that choice in Discussion 9.