Vector database architecture, in depth: index families, memory math, quantization with rescoring and measured recall

What a vector database stores

Each record has an identifier, a dense vector of fixed dimension d, and a payload of ordinary fields (tenant, language, timestamps) used for filtering. The vector usually comes from an embedding model, and a collection must use exactly one model: vectors from two models live in different spaces, and distances between them are meaningless even when the dimensions match.

Distance is one of three functions. Cosine compares direction only. Dot product also rewards magnitude. Euclidean (L2) measures straight-line distance. If you normalise every vector to unit length at write time, the three agree on ranking, because for unit vectors the squared L2 distance equals 2 minus twice the cosine similarity. Normalising once and using dot product is the cheapest option and removes a class of bugs where the query and the index disagree about the metric. Use the metric your embedding model was trained for, as documented by its provider.

Writes follow the same shape in most engines, shown in the diagram above. An upsert goes to a write-ahead log, then into a small growing segment that is searched by brute force. When the segment reaches a size threshold it is sealed and an index is built for it in the background. Deletes are tombstones until compaction rewrites segments. A query fans out to every segment and merges the results. How segment count, compaction and sharding behave at fleet scale is covered in Vector search at scale; here we stay inside one segment's index.

What a vector database stores

Each record has an identifier, a dense vector of fixed dimension d, and a payload of ordinary fields (tenant, language, timestamps) used for filtering. The vector usually comes from an embedding model, and a collection must use exactly one model: vectors from two models live in different spaces, and distances between them are meaningless even when the dimensions match.

Distance is one of three functions. Cosine compares direction only. Dot product also rewards magnitude. Euclidean (L2) measures straight-line distance. If you normalise every vector to unit length at write time, the three agree on ranking, because for unit vectors the squared L2 distance equals 2 minus twice the cosine similarity. Normalising once and using dot product is the cheapest option and removes a class of bugs where the query and the index disagree about the metric. Use the metric your embedding model was trained for, as documented by its provider.

Writes follow the same shape in most engines, shown in the diagram above. An upsert goes to a write-ahead log, then into a small growing segment that is searched by brute force. When the segment reaches a size threshold it is sealed and an index is built for it in the background. Deletes are tombstones until compaction rewrites segments. A query fans out to every segment and merges the results. How segment count, compaction and sharding behave at fleet scale is covered in Vector search at scale; here we stay inside one segment's index.

Flat search: the baseline

The flat index stores vectors contiguously and scans them all. Cost per query is N x d multiply-adds: for one million 768-dimensional vectors, about 768 million, which a modern CPU with SIMD finishes in tens of milliseconds and a GPU in around a millisecond, since the scan reads about 3 GB. Recall is exactly 1.0. Flat search is the right choice for collections up to a few hundred thousand vectors, for the growing segment, and for filtered queries that leave only a few thousand candidates. It is also your ground truth: every approximate index is measured against it.

IVF: partition, then probe

An inverted file index runs k-means on a sample of the vectors to find nlist centroids, then assigns each vector to the list of its nearest centroid. A query compares itself with all centroids, picks the nprobe nearest lists, and scans only those. With 4,096 lists and nprobe 32, a query touches about 0.8% of the data plus 4,096 centroid comparisons.

import numpy as np

def build_ivf(x, nlist, iters=10, seed=0):
    rng = np.random.default_rng(seed)
    cent = x[rng.choice(len(x), nlist, replace=False)]
    for _ in range(iters):                                  # plain k-means
        assign = np.argmax(x @ cent.T, axis=1)              # unit vectors: dot = cosine
        for j in range(nlist):
            members = x[assign == j]
            if len(members):
                c = members.mean(axis=0)
                cent[j] = c / np.linalg.norm(c)
    assign = np.argmax(x @ cent.T, axis=1)
    lists = [np.flatnonzero(assign == j) for j in range(nlist)]
    return cent, lists

def search_ivf(q, x, cent, lists, k=10, nprobe=8):
    probe = np.argsort(-(cent @ q))[:nprobe]                # nearest lists
    ids = np.concatenate([lists[j] for j in probe])
    scores = x[ids] @ q
    top = np.argsort(-scores)[:k]
    return ids[top], scores[top]

IVF's memory overhead is tiny, an id per vector plus the centroids, and its behaviour is easy to reason about: recall rises with nprobe and latency rises linearly with it. Its weakness is the boundary. A true neighbour that sits just across the border of an unprobed cell is missed, so IVF needs a larger nprobe than intuition suggests to reach 0.95 recall. Lists also go stale: if new data drifts away from the training sample, some lists grow huge and queries that hit them slow down, so plan to retrain centroids. A common starting point for nlist is a small multiple of the square root of N; the Faiss project's guidelines suggest between 4 and 16 times the square root.

HNSW and SSD-resident graphs

Hierarchical Navigable Small World graphs, introduced by Malkov and Yashunin, link every vector to a handful of near neighbours and stack sparser layers on top, each node appearing in higher layers with exponentially decreasing probability. A query enters at the top layer, walks greedily towards the query, drops a layer, and repeats; on the bottom layer it runs a best-first search keeping a candidate list of size ef_search. Three parameters control it:

HNSW gives the best recall per millisecond of the in-memory indexes and supports incremental inserts, which is why most engines default to it. The costs are memory and build time. With M of 16, the bottom layer holds 32 neighbour ids of 4 bytes each, 128 bytes per vector, before counting the vectors themselves, which must also be in RAM for distance computations during traversal. Builds are CPU-heavy and take far longer than IVF training at the same scale. Deletions leave holes that degrade connectivity until the graph is repaired or the segment rebuilt. The algorithm itself is explained step by step in HNSW architecture.

SSD-resident graphs, the approach of DiskANN and its Vamana graph, keep compressed codes in RAM to steer the search and store the graph and full vectors on NVMe, reading a few pages per hop. They cut RAM by an order of magnitude at the cost of a few milliseconds of SSD latency per query, which suits collections too large for memory where single-digit-millisecond latency is acceptable.

Quantization, and why it needs rescoring

Quantization stores a lossy code per vector instead of 32-bit floats. It is the main lever on memory and, because distance computations get cheaper, on latency too.

The rule that makes aggressive quantization safe is oversample and rescore: ask the compressed index for more candidates than you need, then recompute exact distances for those candidates with full-precision vectors, which can live on SSD because you only read a few hundred of them per query. Binary codes in particular rank poorly on their own and well as a first stage, and how well they work depends on the embedding model, so test with your own data.

def search_binary_rescore(q, codes, full_vectors, k=10, oversample=10):
    """codes: packed sign bits (uint8 rows); full_vectors: float32, may be memory-mapped."""
    qbits = np.packbits(q > 0)
    hamming = np.unpackbits(codes ^ qbits, axis=1).sum(axis=1)   # cheap first pass
    cand = np.argpartition(hamming, k * oversample)[: k * oversample]
    exact = full_vectors[cand] @ q                                 # few hundred reads
    order = np.argsort(-exact)[:k]
    return cand[order], exact[order]

Worked example: sizing 50 million vectors

Size a collection of 50 million documents embedded at 768 dimensions, with a target of 0.95 recall at k = 10 and a p99 under 50 ms.

Raw float32 vectors take 50,000,000 x 3,072 bytes = 153.6 GB. An HNSW graph with M of 16 adds about 128 bytes per vector on the bottom layer, 6.4 GB, plus a small amount for upper layers. Fully in-memory HNSW therefore needs roughly 160 GB of RAM before replicas, which means several large nodes or shards.

Switch the in-memory vectors to int8 and the vectors shrink to 38.4 GB; with the graph, about 45 GB, which fits on one large node per replica. Keep float32 copies on SSD for rescoring 100 candidates per query, which is 100 random 3 KB reads, well inside NVMe capabilities at moderate query rates. Going further with binary codes, 4.8 GB in RAM plus the graph, is attractive for cost but must be validated: measure recall with oversampling factors of 4, 10 and 20 against flat ground truth before deciding.

The same arithmetic, done for Postgres with halfvec and binary quantization, is worked through in pgvector at 100M scale. Do it before choosing hardware; it is the single most useful page of any vector database design document.

Filtered search in one section

Real queries carry predicates: only this tenant, only English, only documents after a date. Engines combine them with vector search in three ways. Pre-filtering computes the matching ids first and searches only those; when few records match, the engine should switch to a flat scan of them, which Weaviate exposes as a flat-search cutoff. Post-filtering runs the ANN search and drops non-matching results, which silently returns fewer than k results when the filter is selective. In-traversal filtering skips non-matching nodes while walking the graph; with very selective filters the matching nodes may not be connected to each other, so engines add help: Qdrant builds extra graph links between points that share a value of an indexed payload field, and Weaviate's ACORN strategy expands the neighbourhood explored during filtered traversal. Know which strategy your engine uses for each query shape, and test the selective cases explicitly. Engine-by-engine differences are compared in Vector databases compared.

Measuring recall

Recall@k is the fraction of the true k nearest neighbours that the index returned. It depends on the data, the index parameters and the filters, so it must be measured on your data and re-measured whenever any of those change. Keep a fixed sample of real queries, compute ground truth with a flat scan offline, and run the comparison in CI and on a schedule in production.

def recall_at_k(index_search, queries, x, k=10):
    hits = 0
    for q in queries:
        truth = set(np.argpartition(-(x @ q), k)[:k])      # exact top-k
        got = set(index_search(q, k))
        hits += len(truth & got)
    return hits / (k * len(queries))

# sweep the dial and plot latency against recall before picking a setting
for ef in (32, 64, 128, 256):
    r = recall_at_k(lambda q, k: hnsw.search(q, k, ef_search=ef), sample, x)
    print(ef, round(r, 3))

Choose the cheapest setting that meets your recall target on the sweep, then add margin; recall measured on 1,000 queries has a standard error of under one percentage point at these levels, so small differences between settings are often noise. Track recall per filter shape too: a global 0.96 can hide 0.6 for one tenant whose filter is highly selective.

Failure modes

• Model mismatch. Queries embedded with a different model or version than the index return plausible-looking nonsense. Store the model id with the collection and refuse mismatched queries.
• Metric mismatch. Unnormalised vectors indexed for cosine but searched with dot product rank long documents first. Normalise at write time.
• Post-filter starvation. Selective filters return three results instead of ten; users see an empty page. Monitor result counts below k.
• Unindexed growing segments. Heavy ingest outpaces index builds, so more data is scanned flat and latency climbs. Alert on unindexed vector count.
• Tombstone build-up. Frequent updates fill segments and graphs with deleted entries; recall and latency both degrade until compaction. Schedule it and watch the deleted ratio.
• Quantization without rescoring. Recall drops quietly after a memory-saving change. Gate every index change on the recall sweep.
• Re-embedding as an afterthought. Changing models means rebuilding every vector. Keep source text and plan a dual-write migration to a new collection.

Vector database architecture in a distributed cluster

In a distributed vector database, the single-node engine described above is split into roles that scale separately: a stateless access layer, a coordinator, worker nodes for fresh data, historical data and background builds, and a storage layer made of a write-ahead log, a metadata store and object storage. Milvus 2.6 documents this split, and each role maps onto a piece of the design above.

A search follows the same fan-out as in one node. The proxy sends it to the Streaming Nodes that own the collection's shards. Each one searches its growing data, asks Query Nodes for results from sealed segments, and merges them into one result per shard; the proxy then merges the shard results into the final top k. When a flush or compaction produces a sealed segment, the coordinator hands it off to Query Nodes.

The contrast is a single-node engine such as pgvector, where the WAL, the index and the query all run inside one Postgres server: no network hops, and vectors are transactional with the rest of the row, but index memory and build time are bounded by one primary. If the index fits on one node per replica, one process is simpler; see pgvector at 100M scale.

What to do next

1. Write down N, d, metric, k, recall target, latency target and query rate for your collection.
2. Do the memory arithmetic for float32, int8 and binary, with and without the graph, before picking hardware.
3. Normalise vectors at write time and store the embedding model id with the collection.
4. Build a ground-truth set of 1,000 real queries with a flat scan.
5. Sweep ef_search or nprobe and the oversampling factor; pick the cheapest setting that meets recall with margin.
6. Test your most selective filters separately and confirm results never fall below k silently.
7. Alert on unindexed vectors, deleted ratio, result counts below k and scheduled recall.