Title: Lecture 2 — From Text to Vectors: Embeddings, Similarity, and the 2026 Model Landscape
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 02 — Embeddings and Vector Representations for Retrieval

📖 Lecture — From Text to Vectors: Embeddings, Similarity, and the 2026 Model Landscape

Last week you traced the RAG pipeline end to end. This week we zoom into the single component that makes semantic retrieval possible at all: the embedding.

What an embedding is, and why it beats keyword search

An embedding is a dense vector — typically somewhere between 100 and 1000+ dimensions — that represents a piece of text (a word, sentence, paragraph, or chunk). "Dense" is the key word: every dimension carries a small, continuous value, and the model has learned to place semantically similar text close together in that high-dimensional space. This is fundamentally different from a sparse keyword vector (think TF-IDF or a bag-of-words), where each dimension corresponds to a specific vocabulary term and a vector is mostly zeros. Sparse vectors match on shared words; dense embeddings match on shared meaning. A query like "How do I cancel my subscription?" can retrieve a document that says "Steps to terminate your recurring plan" even though the two share almost no vocabulary — because a well-trained embedding model places both near each other in vector space. That proximity property is the entire reason semantic retrieval works, and it is worth internalizing before you touch any vector database, because every downstream decision (chunk size, index type, similarity metric) is in service of preserving and exploiting that proximity.

The indexing pipeline you're building toward

Every production RAG system follows the same four-step sequence, and it's worth naming explicitly now because Weeks 4 and 5 will spend an entire week each on steps 1 and 3:

Step | What happens | Why it matters
1. Segment | Documents are split into chunks (paragraphs, sections, sliding windows) | Chunk size and boundaries directly affect embedding quality — a chunk that's too long dilutes meaning; too short loses context (Week 4)
2. Embed | Each chunk is passed through an embedding model to produce a vector | The model choice made this week determines dimensionality, cost, and retrieval quality for everything downstream
3. Store | Vectors are written to a vector index/database alongside metadata | Index type and database choice affect query latency and filtering ability (Week 3)
4. Query-time embed | The user's query is embedded using the same model used in step 2 | Mismatched embedding models between indexing and query time silently breaks retrieval — there is no error, just bad results

That last row is the most common production bug in RAG systems: someone swaps the embedding model for indexing but forgets to also swap it for the query encoder, or upgrades one but not the other. Because the vectors are still valid numbers, nothing crashes — retrieval quality just quietly degrades. Keep the two paths (index-time and query-time embedding) using the identical model and identical preprocessing, always.

Cosine similarity: why it's standard, and where it breaks

Once you have vectors, you need a way to measure "closeness." Cosine similarity is the default in almost every retrieval system because it is scale-invariant — it measures only the angle between two vectors, ignoring their magnitude. This matters because embedding magnitude often reflects incidental factors (text length, token frequency) rather than meaning, so discarding magnitude and keeping only direction gives a cleaner semantic comparison than, say, Euclidean distance. But cosine similarity is not a magic, objective ruler. Recent research on embedding anisotropy (arXiv:2504.16318) shows that embedding spaces produced by many models are not uniformly distributed — vectors tend to cluster in a narrow cone of the space rather than spreading evenly in all directions. When that happens, cosine similarity between any two vectors — even semantically unrelated ones — can be artificially inflated, simply because they're all pointing in roughly the same general direction. In practice this means a 0.85 cosine similarity score from one model is not directly comparable to a 0.85 from another model, and a high score is not proof of genuine semantic relatedness — it can partly be an artifact of how that particular model's embedding space is shaped.

The 2026 MTEB landscape (and how to read it correctly)

The Massive Text Embedding Benchmark (MTEB) is the closest thing the field has to a standard leaderboard. As of the current MTEB v2 rankings, Gemini Embedding 001 leads with an average score of 68.32, followed by Cohere embed-v4 (65.2) and OpenAI text-embedding-3-large (64.6). On the open-source side, production teams commonly default to BGE-M3, Jina v5-text-small, or Qwen3-Embedding-8B when they need self-hosting, cost control, or data-residency guarantees that a hosted API can't offer. One important housekeeping note: MTEB went through a v1-to-v2 revision, and scores from the two versions are not comparable — if you see an older blog post citing MTEB v1 numbers, don't average them with current v2 numbers or rank models across versions.

Matryoshka representation learning: smaller vectors, almost-free

Storage and query latency both scale with embedding dimensionality, so a 3072-dimension vector costs meaningfully more to store and search than a 256-dimension one — multiplied across millions of chunks, that difference is real infrastructure budget. Matryoshka representation learning solves this elegantly: models trained this way (OpenAI's text-embedding-3-large among them) produce embeddings where the first N dimensions are themselves a valid, useful embedding. You can truncate a 3072-dim output down to 256 dimensions using the dimensions parameter in the API call, and lose only a small amount of retrieval precision while cutting storage and query cost substantially. This is a practical lever you'll return to in Week 3 when we compare vector database costs at scale.

From model choice to database: pgvector

Once you've picked an embedding model and a dimensionality, you need somewhere to put the vectors. The pgvector Python client (pip install pgvector) is the bridge between an embedding decision and a Postgres-based vector column — it lets you store, index, and query embeddings using ordinary Postgres, without standing up a dedicated vector database. We're introducing it here as a preview: next week's lecture will compare pgvector against dedicated vector databases on the tradeoffs that matter for production — scale, filtering, operational overhead, and cost.

Correcting three common misconceptions

"A high cosine similarity score is an objective, model-independent measure of semantic similarity." Not true. Cosine similarity discards magnitude and depends entirely on the geometry the specific model learned during training. Because of anisotropy, even unrelated texts can show inflated similarity scores in some models. Treat cosine similarity as a relative ranking signal within one model's space, never as an absolute, cross-model truth. "Any pretrained model gives good embeddings, and higher dimensionality is always better." Also not true. A model needs to have been trained with a retrieval or contrastive objective (not just general language modeling) to produce vectors that rank well for search. And thanks to Matryoshka training, truncating a high-dimensional embedding down to roughly 256 dimensions typically costs only a few points of precision while cutting storage and latency substantially — so "more dimensions" is not automatically "better return on cost." "The top MTEB leaderboard model is automatically best for your application." MTEB is overwhelmingly a single-language, general-domain retrieval benchmark. It may not reflect your actual language mix, domain vocabulary, or typical document length. The responsible move is to treat the leaderboard as a shortlist generator, then validate candidates against your own documents and queries — which is exactly what this week's lab has you practice.