Title: Discussion 2 — Building a Defensible Embedding-Selection Process
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 02 — Embeddings and Vector Representations for Retrieval

💬 Discussion 2 — Building a Defensible Embedding-Selection Process (20 pts)

This week's lecture and lab both made the same point from different angles: leaderboard rank is not the same thing as "best for your project." Prompt: If MTEB rankings don't guarantee good performance on your own documents, what would a defensible in-house embedding-selection process look like? Consider questions like: What would you actually measure? What data would you test against, and how would you get or simulate it if you don't have production data yet? How would you weigh retrieval quality against cost, latency, and dimensionality (including Matryoshka truncation)? How often would you re-evaluate as new models are released? Post your process (aim for 200+ words) before you read classmates' posts. Then respond to at least two classmates, engaging specifically with a part of their process you agree, disagree, or want to build on — vague "great post!" replies won't earn credit.