Vector Databases & Embeddings

Similarity search is the whole product

A vector database stores embeddings, high-dimensional numeric vectors (typically 384 to 3072 dimensions) produced by an embedding model, and answers similarity search: "give me the K stored vectors closest to this query vector." This is the retrieval backbone of semantic search, recommendations, deduplication, and RAG (retrieval-augmented generation), where you embed a user's question, find the most similar document chunks, and feed them to an LLM as context. The whole value is that "closeness" in embedding space approximates semantic meaning, so a query for "how do I reset my password" retrieves a chunk about "recovering account access" even with zero shared keywords.

Why not exact search. Finding the true nearest neighbors means comparing the query against every stored vector, O(N x d). At a few thousand vectors that is fine; at millions or billions it is far too slow for an interactive query. So vector databases use ANN (approximate nearest neighbor) search: give up a small amount of recall (you might miss a few of the true top-K) in exchange for orders-of-magnitude faster queries. The central tradeoff of the whole family is recall vs latency vs memory, and picking an index is picking a point on that surface.

The index families you must know

HNSW (Hierarchical Navigable Small World) builds a layered graph where each vector links to nearby vectors; search greedily hops through the graph from a coarse top layer down to a fine bottom layer. It gives high recall at low latency and is the default for most workloads. The cost is memory: the graph and vectors live in RAM, so it is expensive at billion scale. Tunable knobs: M (links per node) and efSearch (candidates explored, higher = better recall, slower).

IVF (Inverted File) clusters vectors into nlist partitions (via k-means) and, at query time, only searches the few nearest partitions (nprobe). More memory-efficient and faster to build than HNSW but lower recall unless you probe more partitions. PQ (Product Quantization) compresses each vector into a short code (e.g. a 1536-dim fp32 vector is 1536 x 4 = 6144 bytes; PQ codes it in 64 bytes, a 96x reduction), slashing memory 10 to 100x at the cost of some recall. IVF-PQ combines them and is the go-to for billion-scale, memory-constrained deployments (what FAISS is known for). Rule of thumb: HNSW when recall and latency matter and you can afford RAM; IVF-PQ when scale and memory dominate.

Two essentials beyond raw similarity

Metadata filtering. Real queries are "similar chunks from this user's documents, in English, updated this year." You store metadata alongside each vector and filter on it. The subtlety is pre-filter vs post-filter: post-filtering (find top-K by vector, then drop non-matching) can return too few results if the filter is selective; good systems do filtered ANN that respects the filter during traversal. Ask about this; it is a common gotcha.

Hybrid search. Pure vector search misses exact keyword matches (product codes, names, rare terms). Hybrid search combines vector similarity with a keyword/BM25 lexical score, fused (often via reciprocal rank fusion), giving both semantic recall and lexical precision. Production RAG almost always uses hybrid.

Interview nuance: "pgvector or a dedicated vector store?" pgvector (Postgres extension) is the right call when your corpus is modest (up to low millions), you already run Postgres, and you want vectors next to relational data and transactions with no new system. Reach for a dedicated store (Pinecone, Weaviate, Qdrant, Milvus) at tens of millions to billions of vectors, when you need advanced filtered ANN, horizontal scaling, or managed operations. Do not add a specialized vector database for 50k chunks; pgvector is plenty.

Design choices that bite later: the embedding model fixes your dimensionality and distance metric (cosine for normalized text embeddings, dot product, or L2). Chunking strategy (size and overlap) hugely affects retrieval quality. And critically, re-embedding migrations: if you switch embedding models, every stored vector is now in a different space and must be re-embedded, an expensive backfill you must plan for, so version your embeddings.

Recap: Vector databases do approximate nearest-neighbor search over embeddings, trading recall for latency and memory; choose HNSW for recall at cost of RAM or IVF-PQ for billion-scale memory efficiency, always add metadata filtering and hybrid (vector + BM25) search, use pgvector until scale forces a dedicated store, and plan for re-embedding when the model changes.