By now you've fine-tuned a model and quantized one. This week we look at something that seems like a formatting detail but is actually a common cause of fine-tunes quietly underperforming: the chat template.
When you talk to a chat model, you hand it a conversation — a list of turns, each with a role (system, user, assistant, sometimes tool) and content. The model itself doesn't understand Python dictionaries; it only understands a single string of tokens. Something has to convert the structured conversation into that string, inserting markers so the model knows where the system prompt ends, where the user's turn starts, and where it should begin generating. That "something" is the chat template — a Jinja template stored as metadata on the tokenizer (in tokenizer_config.json as chat_template). Calling tokenizer.apply_chat_template(messages, tokenize=False) runs your {role, content} dicts through that template and returns the exact formatted string the model was trained to expect — control tokens, whitespace, turn boundaries, and all. Set tokenize=True (the default) and it also tokenizes the result in the same step. This function bridges "messages you can reason about" and "the string the model actually sees."
Every model family defines its own control tokens, and the Jinja logic assembling them is family-specific. Mistral-style models wrap the user turn in [INST]/[/INST]. Zephyr-style models use <|user|>/<|assistant|> on their own lines. Some newer models — especially agent or RAG models — expose multiple named templates on one tokenizer: plain chat, tool-calling, and RAG context injection.
| Model family | Example turn markers | Notes | ||||
|---|---|---|---|---|---|---|
| Mistral / Mixtral-Instruct | [INST] ... [/INST] |
No explicit system-turn tag originally | ||||
| Zephyr / many HF fine-tunes | \\<</td>
<td>system</td>
<td>\>, \<</td>
<td>user</td>
<td>\>, \<</td>
<td>assistant</td>
</tr>
<tr>
<td>Llama-3 / Llama-3.1</td>
<td>\\< |
start_header_id | \>role\< | end_header_id | \>\</td>
<td>Header tokens plus \\< |
|
| Qwen2.5 | \\<</td>
<td>im_start</td>
<td>\>role, \<</td>
<td>im_end</td>
<td>\>\ |
ChatML-style formatting |
You cannot borrow the Zephyr template for a Llama-3 model. The wrong template produces a string the model was never trained to parse — it may still generate something, but the model's sense of turn structure breaks down.
The key rule this week: the chat template used at fine-tuning time must exactly match the template the base model originally saw during its own instruction-tuning. A mismatch doesn't usually throw a loud error — you get silent degradation. Loss still goes down, but the model responds worse, ignores system prompts, runs on past the intended stop point, or produces malformed turns. Because nothing crashes, this is a common unexplained-failure pattern in fine-tuning.
TRL's SFTTrainer was built with this problem in mind. It natively accepts conversational datasets — rows that are lists of {role, content} dicts — and during preprocessing automatically calls the tokenizer's own apply_chat_template() on every example. You no longer hand-assemble prompt strings and hope you got the special tokens right; the trainer defers to the one source of truth for what the model expects.
If you introduce a custom chat template that adds control tokens the base model has never seen (a new <|tool_call|> marker, say), those tokens must be registered as special tokens and the embedding matrix resized so it has a vector for them. A base model does not automatically understand a token it has never trained on — its embedding starts uninitialized or random, and needs fine-tuning exposure to learn a meaningful representation. Skipping this step is another quiet failure: the token exists in the vocabulary but carries no learned meaning.