🛠️ Lab 4 — Chat Template Detective: Inspect, Compare, and Preprocess (50 pts)

Goal: Get hands-on proof of how chat templates convert {role, content} messages into model-ready strings, see why templates differ across model families, and practice preprocessing a real dataset with apply_chat_template() as you would before fine-tuning. Steps:

1. Open a free Google Colab notebook (no paid tier or local GPU required for this lab). Install transformers, datasets, and trl.
2. Load the tokenizer for a Llama-3 model (e.g., meta-llama/Meta-Llama-3-8B-Instruct) with AutoTokenizer.from_pretrained(...). (If a gated repo blocks you, use any publicly available Llama-3-family tokenizer or substitute a comparable open tokenizer — the point is the template, not the specific weights, and you never need to download the full model to inspect its tokenizer.)
3. Build a small list of {"role": ..., "content": ...} message dicts representing a short conversation (system + user + assistant turns).
4. Call tokenizer.apply_chat_template(messages, tokenize=False) and print the raw output string. Annotate in your notebook (as markdown comments) where the turn boundaries and special tokens appear.
5. Load a second tokenizer from a different model family (e.g., Qwen2.5) and repeat step 4 with the same message list. Print both outputs side by side.
6. Write a short comparison (in a markdown cell) of the two outputs: which control tokens differ, how turn boundaries are marked, and one concrete reason you could not swap one template for the other without breaking the model's expectations.
7. Load the databricks/databricks-dolly-15k dataset from the Hugging Face Hub. Take one raw example and manually rewrite it as a chat-style message list (system/user/assistant, using the Dolly instruction, context, and response fields).
8. Now preprocess an entire split of the dataset (a small subset, e.g., .select(range(200)), is fine for compute reasons) using .map() to convert each row into a message list and apply the Llama-3 tokenizer's apply_chat_template() with tokenize=True.
9. Compute and print basic tokenized-length statistics for the preprocessed split (min, max, mean token length) to confirm your preprocessing step ran correctly before you'd ever hand it to a trainer.

Deliverables: Submit your Colab notebook (.ipynb, shared link or file upload) containing all code cells with output visible, your side-by-side template comparison writeup, and your tokenized-length statistics, plus a 150–250 word reflection on what would happen if you fed the Qwen-formatted string into the Llama-3 tokenizer's training pipeline by mistake.