Title: Lecture 4 — Chat Templates, Tokenization, and Why Formatting Breaks Fine-Tunes
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 04 — Instruction and Chat Formatting, Tokenization, and Templates

📖 Lecture — Chat Templates, Tokenization, and Why Formatting Breaks Fine-Tunes

By now you've fine-tuned a model and quantized one. This week we look at something that seems like a formatting detail but is actually a common cause of fine-tunes quietly underperforming: the chat template.

What a chat template actually is

When you talk to a chat model, you hand it a conversation — a list of turns, each with a role (system, user, assistant, sometimes tool) and content. The model itself doesn't understand Python dictionaries; it only understands a single string of tokens. Something has to convert the structured conversation into that string, inserting markers so the model knows where the system prompt ends, where the user's turn starts, and where it should begin generating. That "something" is the chat template — a Jinja template stored as metadata on the tokenizer (in tokenizer_config.json as chat_template). Calling tokenizer.apply_chat_template(messages, tokenize=False) runs your {role, content} dicts through that template and returns the exact formatted string the model was trained to expect — control tokens, whitespace, turn boundaries, and all. Set tokenize=True (the default) and it also tokenizes the result in the same step. This function bridges "messages you can reason about" and "the string the model actually sees."

Templates are not interchangeable

Every model family defines its own control tokens, and the Jinja logic assembling them is family-specific. Mistral-style models wrap the user turn in [INST]/[/INST]. Zephyr-style models use <|user|>/<|assistant|> on their own lines. Some newer models — especially agent or RAG models — expose multiple named templates on one tokenizer: plain chat, tool-calling, and RAG context injection.

Model family | Example turn markers | Notes |  |  |  |
Mistral / Mixtral-Instruct | [INST] ... [/INST] | No explicit system-turn tag originally |  |  |  |
Zephyr / many HF fine-tunes | \\<</td> <td>system</td> <td>\>, \<</td> <td>user</td> <td>\>, \<</td> <td>assistant</td> </tr> <tr> <td>Llama-3 / Llama-3.1</td> <td>\\< | start_header_id | \>role\< | end_header_id | \>\</td> <td>Header tokens plus \\<
Qwen2.5 | \\<</td> <td>im_start</td> <td>\>role, \<</td> <td>im_end</td> <td>\>\ | ChatML-style formatting

You cannot borrow the Zephyr template for a Llama-3 model. The wrong template produces a string the model was never trained to parse — it may still generate something, but the model's sense of turn structure breaks down.

Why mismatches are dangerous during fine-tuning

The key rule this week: the chat template used at fine-tuning time must exactly match the template the base model originally saw during its own instruction-tuning. A mismatch doesn't usually throw a loud error — you get silent degradation. Loss still goes down, but the model responds worse, ignores system prompts, runs on past the intended stop point, or produces malformed turns. Because nothing crashes, this is a common unexplained-failure pattern in fine-tuning.

How TRL's SFTTrainer helps

TRL's SFTTrainer was built with this problem in mind. It natively accepts conversational datasets — rows that are lists of {role, content} dicts — and during preprocessing automatically calls the tokenizer's own apply_chat_template() on every example. You no longer hand-assemble prompt strings and hope you got the special tokens right; the trainer defers to the one source of truth for what the model expects.

New tokens need to learn their embeddings

If you introduce a custom chat template that adds control tokens the base model has never seen (a new <|tool_call|> marker, say), those tokens must be registered as special tokens and the embedding matrix resized so it has a vector for them. A base model does not automatically understand a token it has never trained on — its embedding starts uninitialized or random, and needs fine-tuning exposure to learn a meaningful representation. Skipping this step is another quiet failure: the token exists in the vocabulary but carries no learned meaning.

Correcting this week's misconceptions

- "Chat templates are a minor formatting detail." They are not. A template/tokenization mismatch is a leading cause of silent degradation, even during continued fine-tuning of an already-instruction-tuned model. Verify the template before training, not after the model behaves oddly.
- "Any base model already understands new chat-template special tokens." False. New tokens from a custom template usually have no trained embedding. Add them as special tokens, resize the embedding layer, and let fine-tuning teach the model what they mean.