So far in this program you have used pretrained models as-is: prompting them, wrapping them in containers, serving them behind an API. This week we start actually changing model weights. The entry point for nearly every real-world adaptation project is Supervised Fine-Tuning (SFT) — and once you understand it, everything you learn later about preference tuning, LoRA, and quantization will build directly on this foundation.
SFT is conceptually simple: you show the model paired input/output sequences — a prompt and a desired completion — and train it with the same objective it already knows from pretraining, next-token prediction. The only thing that changes is what it's predicting: instead of predicting the next token of arbitrary internet text, it's predicting the next token of a completion you have chosen as correct or desirable. Repeated over thousands of examples, the model's weights shift toward producing that style and structure of completion when it sees similar prompts. This is why SFT is the right starting point for adaptation. It doesn't require a reward model, a preference dataset, or reinforcement learning machinery — just examples of the behavior you want. If you can write down (or collect) "given this input, here is the output I want," you can fine-tune for it.
Hugging Face's TRL (Transformer Reinforcement Learning) library ships a purpose-built SFTTrainer that removes nearly all of the boilerplate. A working fine-tune can be as short as:
from trl import SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B",
train_dataset=dataset,
)
trainer.train()That's it — no manual tokenizer setup, no manual chat-template application, no manual batching logic. SFTTrainer inspects the dataset, applies the correct chat template if the data is conversational, tokenizes everything, and (as covered below) packs sequences efficiently. This is a deliberate design choice by the TRL team: the defaults are tuned to be correct for instruction tuning out of the box, so you can focus on your data and hyperparameters rather than plumbing.
A common assumption newcomers make is that they need to hand-build prompt strings themselves, concatenating a system message, user turn, and assistant turn into one blob of text before handing it to the trainer. This is not necessary, and it's a misconception worth correcting directly. TRL's SFTTrainer natively accepts conversational datasets formatted as lists of role/content dictionaries:
{"messages": [
{"role": "user", "content": "What is quantization?"},
{"role": "assistant", "content": "Quantization reduces the numeric precision..."}
]}Given this format, SFTTrainer automatically applies the model's chat template (the special tokens and turn-formatting the base model expects) before tokenizing. You only need to hand-build prompt/completion strings if you are working with a plain prompt-completion dataset format rather than a conversational one — and even then, TRL handles the tokenization and masking for you.
| Dataset format | Structure | Who applies the chat template? |
|---|---|---|
| Conversational | {"messages": [{"role":..., "content":...}, ...]} |
TRL, automatically |
| Prompt-completion | {"prompt": "...", "completion": "..."} |
You control the raw text directly; TRL still tokenizes and masks |
Here is the second major misconception to correct: loss is not computed over the entire sequence. If it were, the model would be trained to predict the prompt tokens too — effectively learning to reproduce questions and instructions rather than focusing all its gradient signal on producing good answers. That would waste capacity and dilute the training signal. By default, SFTTrainer sets completion_only_loss=True, which builds a completion mask so that only completion tokens contribute to the loss — prompt tokens are masked out entirely. This is the correct, standard behavior for instruction tuning, and it's why TRL enables it by default rather than requiring you to opt in. You can set completion_only_loss=False to compute loss over the full sequence, but this is not recommended for instruction-tuning workflows; you would only consider it in unusual cases such as pure language-modeling continuation training where there's no meaningful prompt/completion split.
When your dataset is too large to fit comfortably in RAM, SFTTrainer supports Hugging Face IterableDataset. Instead of materializing the whole dataset up front, TRL tokenizes and packs examples on the fly as training consumes them. For this course, our lab datasets are small enough to load normally, but you should know this option exists — it's the difference between a fine-tune that works on your laptop and one that runs out of memory before training even starts.
Two mechanisms make TRL's default training loop efficient rather than just correct:
chunked_nll loss, made the default in TRL v1.7, computes the negative log-likelihood loss in chunks rather than materializing the full logits tensor at once. This cut peak VRAM usage by roughly 30% on average — a meaningful difference when you're working on a free-tier T4 GPU with limited memory headroom.| Mechanism | What it solves | Typical impact |
|---|---|---|
| Sequence packing | Wasted compute on padding for short examples | \~3–5x less wasted computation |
chunked_nll (default since TRL v1.7) |
Peak VRAM from materializing full logits | \~30% average VRAM reduction |
completion_only_loss=True (default) |
Gradient signal diluted by prompt tokens | Correct, focused instruction-tuning signal |
Taken together, these defaults mean that a beginner running SFTTrainer out of the box on a small model and a conversational dataset is already getting sensible masking, efficient memory use, and reduced padding waste — without touching a single configuration flag. Understanding why those defaults exist is what will let you debug things confidently when a training run doesn't behave the way you expect.