Lecture 1 — Choosing Your Customization Path: Prompting, RAG, or Fine-Tuning

📖 Lecture — Choosing Your Customization Path: Prompting, RAG, or Fine-Tuning

Welcome to AIINFRA 201. In 101 and 102 you learned to build with large language models as they come out of the box — prompting them, wiring them into containers, and calling their APIs. Starting this week, we tackle a harder question: what do you do when the out-of-the-box model isn't good enough? Maybe it doesn't know your company's product catalog. Maybe it answers in the wrong tone. Maybe it can't reliably return the exact JSON shape your downstream system needs. The instinct many teams have is to reach immediately for fine-tuning — retraining the model on custom data. This week's central lesson is that fine-tuning should almost never be your first move. The canonical decision sequence. When a model isn't performing the way you need, work through these options in order:

Prompt engineering first. Rewrite your instructions, add examples (few-shot prompting), restructure your system prompt, or add formatting constraints. This is the cheapest and fastest option — changes take minutes, cost nothing beyond normal API calls, and require no infrastructure.
Add retrieval-augmented generation (RAG) if knowledge grounding is the problem. If the model doesn't know something — your latest pricing, a policy that changed last week, internal documentation — RAG retrieves the relevant text at query time and hands it to the model as context. This takes days to build a first version, not months.
Fine-tune only when behavior or style must be baked in. Fine-tuning means updating the model's weights on your own examples so a behavior becomes intrinsic rather than something you have to re-explain in every prompt. This is the slowest and most expensive path, and it should be your last resort, not your first instinct.

The reason for this order is simple economics: each step up costs more in engineering time, compute, and ongoing maintenance. Most problems that look like "the model needs to be fine-tuned" are actually solved by a better prompt or a retrieval step. RAG and fine-tuning solve different problems. It helps to think of them as addressing two separate axes of a model's performance:

Dimension	RAG	Fine-tuning
What it changes	What the model knows at answer time	How the model behaves
Best for	Fresh facts, citations, frequently changing information	Tone, brand voice, structured output consistency, domain-specific reasoning patterns
Update speed	Update the document store anytime; no retraining	Requires a new training run to change behavior
Cost profile	Ongoing retrieval/vector-search infra cost	Upfront training cost, cheaper per-inference at scale
Failure mode if misused	Irrelevant retrieved context, weak grounding	Stale facts baked into weights, hallucination risk

When fine-tuning is actually the right call. There are legitimate reasons to fine-tune, and you should recognize them:

You need a durable tone or brand voice that must show up consistently, not just when you remember to specify it in a prompt.
You need structured output that prompting can't reliably guarantee — for example, a JSON schema that must never break even under adversarial or unusual inputs.
Latency constraints rule out a retrieval step, so the behavior has to be intrinsic to the model rather than assembled at inference time.
High volume makes a smaller, fine-tuned open-weight model dramatically cheaper to run than repeated frontier API calls, even after you pay the fine-tuning cost upfront.

The hybrid reality. In production, the choice is rarely "RAG or fine-tuning" — it's usually both. A well-built support chatbot might use RAG to pull the current return policy and product specs (so answers are accurate and citable) while relying on a lightly fine-tuned model to keep every response in the company's voice and in a consistent structured format the UI can render. RAG and fine-tuning are complementary tools addressing different failure modes, not competitors for the same job. Correcting the misconceptions. Two mistaken beliefs are common enough that we want to name them directly:

"Fine-tuning is the default or first choice for model adaptation." It is not. It is the most expensive and slowest option and should only be reached for after prompt engineering and RAG have been genuinely exhausted. Teams that reach for fine-tuning first usually end up rebuilding the same solution with a simpler prompt or retrieval pipeline once they discover the maintenance burden.
"Fine-tuning can efficiently teach a model new facts." Research on this is unambiguous: when fine-tuning examples introduce facts the base model didn't already know, the model learns them slowly and inconsistently, and — critically — the presence of unfamiliar facts in the training data linearly increases the model's hallucination rate on unrelated questions later. If your problem is "the model doesn't know X," the correct tool is RAG, which injects that knowledge at query time without touching the model's weights at all. Fine-tuning is a tool for teaching behavior, not for teaching facts.

Keep this mental model as you move through the rest of the course: prompt engineering changes what you ask, RAG changes what the model can see, and fine-tuning changes who the model is. Match the tool to the actual problem, and you'll save yourself weeks of unnecessary training runs.