Vector Databases

Vector databases went from niche to ubiquitous on the back of the LLM boom, because they answer a question keyword search can't: "find me things that mean the same, not just things that share the same words." They store high-dimensional embeddings and find the nearest ones to a query vector fast, which is the retrieval half of RAG (retrieval-augmented generation) — the pattern behind most production LLM apps, including the context assembly we discussed in harness engineering and building with LLM APIs.

Embeddings: Meaning as Geometry

An embedding is a fixed-length vector (say 768 or 1536 numbers) produced by a model from a piece of content. The model is trained so that semantically similar inputs map to nearby vectors: "dog" and "puppy" land close together; "dog" and "tax form" land far apart. This works across modalities — text, images, audio — and it's what lets you search by meaning rather than exact words. The vector itself is opaque (no single dimension "means" anything human-readable), but distances between vectors are meaningful, and that's all search needs.

Similarity Metrics

"Nearest" requires a distance measure. The common ones:

For most text use cases the vectors are normalized and cosine/dot-product are effectively equivalent. The key point for interviews: similarity search ranks by one of these metrics, returning the k closest vectors.

The Core Problem: Approximate Nearest Neighbor

The naive way to find the nearest vectors is to compute the distance from the query to every stored vector and keep the top k — O(n·d) where n is the number of vectors and d the dimensions. At millions or billions of vectors, with d in the hundreds, that's far too slow for interactive search. So vector databases use approximate nearest neighbor (ANN) algorithms that find almost the true nearest neighbors in sublinear time. The trade is recall (what fraction of the true top-k you actually return) for speed — and well-tuned ANN reaches ~95–99% recall while being orders of magnitude faster.

ANN Algorithms: HNSW and IVF

Two families dominate:

• HNSW (Hierarchical Navigable Small World) — builds a multi-layer graph where each vector links to its near neighbors. Search starts at a sparse top layer and "greedily" hops toward the query, descending layers to refine. It gives excellent recall and speed, and is the default in most vector DBs; the cost is higher memory and build time.
• IVF (Inverted File Index) — clusters vectors (e.g. via k-means) into buckets; a query only searches the few clusters nearest its vector instead of all of them. Cheaper memory, slightly lower recall, tunable by how many clusters you probe.

layer 2 (sparse):   ●───────────●          start here, hop toward query
                    │           │
layer 1:        ●──●──●─────●───●──●        descend, refine
                │  │  │     │   │  │
layer 0 (all):  ●●●●●●●●●●●●●●●●●●●●●        finest neighbors → top-k

  each step: move to the neighbor closest to the query vector
  → finds ~nearest neighbors in O(log n)-ish hops, not O(n)

To cut memory, vectors are often quantized — e.g. product quantization (PQ) compresses each vector into a compact code, trading a bit more recall loss for a large memory reduction. Real systems combine these (IVF + PQ, or HNSW + PQ).

What a Vector Database Adds

You can run an ANN library (FAISS, hnswlib) in-process, so why a database? Because production needs more than the index:

• Metadata filtering — "nearest vectors where tenant = X and date > Y." Combining filters with ANN correctly is non-trivial and a core vector-DB feature.
• Persistence & CRUD — durably store, update, and delete vectors (re-indexing on change), not just a static in-memory index.
• Scaling & availability — sharding across nodes, replication, and horizontal scale.
• Operations — backups, monitoring, access control.

Options span dedicated systems (Pinecone, Weaviate, Milvus, Qdrant) and extensions to existing databases (pgvector for PostgreSQL, vector search in Elasticsearch/Redis) — the latter attractive when you don't want a new datastore.

RAG: the Killer Use Case

Retrieval-augmented generation is why vector DBs exploded. An LLM has a fixed context window and no knowledge of your private data, so RAG retrieves the relevant pieces and puts them in front of the model:

INDEX (offline):
   docs → chunk → embed each chunk → store vectors + metadata

QUERY (online):
   user question → embed → ANN search → top-k relevant chunks
                → stuff chunks into the LLM prompt as context
                → LLM answers grounded in the retrieved text

This is exactly the "context assembly" job from harness engineering: the vector DB is how the harness finds the right chunks to put in a finite context window. Chunking strategy and retrieval quality matter as much as the model.

Hybrid Search and vs Elasticsearch

Pure semantic search can miss exact matches (a specific product code, a rare name) that lexical search nails, and vice versa. Hybrid search combines vector similarity with keyword (BM25) scoring and fuses the rankings, usually beating either alone. This is where vector DBs and Elasticsearch converge: classic Elasticsearch is lexical, vector DBs are semantic, and both are adding the other so that hybrid is becoming the norm.

Pitfalls

• Recall tuning — ANN parameters trade recall for speed/memory; the defaults aren't always right for your accuracy needs.
• Embedding consistency — query and documents must use the same embedding model; changing models means re-embedding everything.
• Cost & dimensions — high-dimensional vectors at scale are memory-hungry; quantization and dimension choice matter.
• Freshness & deletes — graph indexes like HNSW handle deletes/updates awkwardly; heavy churn may need periodic rebuilds.