Title: Lab 2 — Benchmark Two Embedding Models with MTEB
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 02 — Embeddings and Vector Representations for Retrieval

🛠️ Lab 2 — Benchmark Two Embedding Models with MTEB (50 pts)

Goal: Practice justifying an embedding model choice with your own evidence instead of leaderboard marketing, by running the free, open-source mteb and sentence-transformers libraries locally (or on free Google Colab) to compare two open embedding models on a real retrieval task. Steps:

1. Set up your environment. On your own machine or in a free Colab notebook, install the required libraries: pip install mteb sentence-transformers.
2. Load your first model. Use sentence-transformers/all-MiniLM-L6-v2 as your baseline — it's small, free, and runs comfortably on CPU.
3. Select a small retrieval task. Use mteb.get_tasks() to browse available tasks and choose one small retrieval task (filter by task type "Retrieval" and pick one with a modest corpus size so it runs quickly on free compute).
4. Run the benchmark. Call mteb.evaluate() (or MTEB(tasks=[...]).run(model), depending on your installed mteb version — check the version's docs) against your chosen task and record the resulting score(s), including which metric(s) were reported (e.g., NDCG@10).
5. Repeat with a second open model. Choose a different open embedding model available on Hugging Face (for example, a BGE, Jina, or Qwen3-Embedding variant that fits on your available compute) and run the exact same task and evaluation code so the comparison is apples-to-apples.
6. Compare and interpret. Put both models' scores side by side. Note the runtime/resource cost of each run, and any qualitative differences you noticed (e.g., one model timing out, needing more memory, or producing very different score patterns).
7. Write your justification. In 300–500 words, explain which of the two models you would recommend for a hypothetical production use case of your choosing, using your own benchmark numbers — not the public MTEB leaderboard rank — as your primary evidence. Explicitly address at least one limitation of your own mini-benchmark (e.g., single task, small corpus, no domain match to production data).

Deliverables: Submit (a) your notebook or script with both evaluation runs and their output, and (b) your 300–500 word written justification, either pasted into the text entry box or as an uploaded file (PDF, Markdown, or .ipynb).