Title: Lab 4 — Build, Benchmark, and Filter: HNSW vs IVF Head-to-Head
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 04 — Indexing Deep Dive: HNSW vs IVF and Metadata Filtering

🛠️ Lab 4 — Build, Benchmark, and Filter: HNSW vs IVF Head-to-Head (50 pts)

Goal: Build both an IVFFlat and an HNSW index over the same dataset in local Postgres/pgvector, measure build time, memory footprint, recall, and query latency for each, then extend the comparison into Qdrant by adding metadata (payload) filtering and observing how filtering changes the recall-vs-latency-vs-index-type relationship you learned about this week.

Steps

1. Set up your environment. Using free/local tools, install PostgreSQL with the pgvector extension (via Docker: docker run -d --name pg-vec -e POSTGRES_PASSWORD=postgres -p 5432:5432 ankane/pgvector, or a local Postgres + CREATE EXTENSION vector;). Also install Qdrant locally (docker run -d --name qdrant -p 6333:6333 qdrant/qdrant).
2. Load a dataset. Reuse the document/embedding set from a prior week's lab (or generate at least 5,000–10,000 embeddings using a free local model, e.g., sentence-transformers/all-MiniLM-L6-v2) and load them into a Postgres table with an embedding vector(384) column (adjust dimension to your model) plus a doc_type metadata column with at least 2–3 categories.
3. Build an IVFFlat index and measure it. Time the build:

CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops) WITH (lists = 100);

Record wall-clock build time and check the index size with \di+ or pg_relation_size.

1. Build an HNSW index and measure it. On a fresh copy of the same table (or after dropping the IVFFlat index), time:

CREATE INDEX ON items USING hnsw (embedding vector_l2_ops) WITH (m = 16, ef_construction = 64);

Record build time and index size, and compare against IVFFlat.

1. Measure recall and latency for both. Pick 20–50 query vectors. For each index, run queries at multiple settings — IVFFlat at probes = 1, 5, 10 and HNSW at hnsw.ef_search = 20, 40, 100 — and compute recall against a brute-force (unindexed sequential scan) ground truth, along with average query latency at each setting. Record results in a table.
2. Extend to Qdrant with metadata filtering. Load the same vectors plus their doc_type payload field into a Qdrant collection. Create a payload index on doc_type. Build an HNSW index (Qdrant's default) and run: (a) unfiltered searches, and (b) filtered searches restricting to a doc_type that represents roughly 10% of the corpus (i.e., a highly selective, \>90%-excluding filter).
3. Sweep hnsw_ef under filtering. In Qdrant, sweep the query-time hnsw_ef parameter (e.g., 20, 40, 100, 200) for both the unfiltered and filtered searches. Record recall and latency at each setting for both conditions.
4. Write up your findings. Summarize, in your own words, what you observed: How did HNSW and IVFFlat compare on build time and memory in Postgres? How did recall/latency change as you raised nprobe/ef_search? Most importantly — how did latency and its variance change once you added the selective metadata filter in Qdrant, and does that match what the lecture predicted about HNSW degrading under heavy filtering?

Deliverables:

Submit (a) your SQL/Python/script files used to build indexes, run benchmarks, and query Qdrant, (b) a results table (CSV or Markdown) covering build time, index size, recall, and latency for every configuration tested, and (c) a short written summary (300–500 words) answering the analysis questions in Step 8, uploaded as a single document or zip via online upload, or pasted as text.