Evidence label: deterministic synthetic model. This lab runs a small JavaScript model of quantized vector search. It generates 1,000 clustered synthetic vectors with 128 dimensions and 32 noisy queries, finds the exact top 10 for each query with float dot products, then repeats the search with int8, int4 or binary codes built from the same vectors, with an optional rescoring stage that re-ranks a shortlist with the original floats. Memory, latency and cost figures are calculated for a 1024-dimensional index of the size you choose. Nothing here calls a vector database or measures hardware.
Play the lab
How to play
Choose an encoding, a rescoring mode and an oversampling factor, then set the corpus size and node memory. Every change re-runs the full synthetic search with the same seed, so differences come from your change. The tiles show recall@10 against exact float search, the RAM the index needs, an estimated query latency, the monthly node and SSD cost, the compression ratio and the SSD reads each query makes. Use the events to inject outlier spikes, switch to a non-centred embedding model or triple the corpus. Each mission loads a broken setup; fix it, then try for three stars with the cheapest configuration that meets the goal.
What the model assumes
Vectors are drawn from 16 Gaussian clusters with a different spread per dimension, then normalized. Queries are copies of random documents with small noise added. Int8 and int4 use one global range, either the full min/max or a 99.9% quantile with clipping; the query stays in float. Binary codes threshold each dimension at zero or at its corpus mean and compare with Hamming distance. Memory assumes 1024 dimensions, 128 bytes of HNSW links per vector and 10% node headroom; latency is 1 ms of graph overhead plus 2,000 distance computations at a per-encoding cost, 0.06 ms per rescored candidate read from SSD, and a large penalty for the share of the index that does not fit in RAM. RAM is priced at an illustrative $4 per GB-month and SSD at $0.08.
What it cannot tell you
Real embeddings have 384 to 3,072 dimensions and structure that synthetic clusters only imitate; binary codes in particular work better at higher dimension than in this 128-dimensional model. The lab searches exhaustively rather than through an HNSW graph, so it ignores the recall lost by approximate graph search and the interaction between quantization and graph construction. Latency and cost constants are illustrative. Use the lab to understand the shape of the trade-offs, then measure recall and latency on your own data and engine version.
Read the full explanation
- Embedding Quantization: Binary and Int8 Vectors Without Losing Recall (companion article)
- HNSW Tuning for RAG: M, efSearch and the Recall Versus Latency Trade
- Cross-Encoder Reranking Depth: How Many Candidates Are Worth the Latency
All AI in Production labs · Lab catalog
- Hugging Face blog: Binary and Scalar Embedding Quantization for Significantly Faster and Cheaper Retrieval
- Qdrant documentation: Quantization
- pgvector README (halfvec, bit, binary_quantize)
- Elasticsearch reference: dense_vector field type
- RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search (SIGMOD 2024)
- The Faiss library (2024)
Views are my own and do not represent any employer, past or present.