Why architecture matters here
Vector database architecture matters because vector search has fundamentally different performance characteristics than exact-match search. Nearest-neighbor is approximate; index build times can be hours; recall trade-offs are measured, not axiomatic. Choosing the right index and tuning it correctly can be the difference between 5 ms and 500 ms per query at the same recall.
Cost matters. Vector data is dense — 1024 float dimensions per document is 4 KB, so a million documents is 4 GB just for vectors. Quantization shrinks that 4-8x with modest recall loss. Sharding scales beyond single-node.
Reliability requires attention. Rebuilding a HNSW index is expensive; safely rebalancing sharded vector data is nontrivial. The operational discipline matters as workloads grow.
The architecture: every piece explained
Walk the diagram top to bottom.
Documents. Text, images, audio embeddings, or any data with a vector representation. Each document has an ID and metadata (tags, timestamps, tenant).
Embedder. An embedding model (text-embedding-3, BGE, Voyage) that turns content into a fixed-length vector. Batching for throughput.
Vector DB. Stores vectors + metadata + index. Serves nearest-neighbor queries.
Metadata Filter. Structured predicates applied to queries: "only from tenant X" or "created after 2026-06." Vector DBs vary widely in how efficiently they combine filters with vector search.
Hybrid Search. Combine sparse (BM25 keyword) and dense (vector) rankings. Fusion via reciprocal rank or weighted score. Consistently improves recall on real queries.
Index Structure. HNSW (hierarchical navigable small world) — graph-based, fast queries. IVF (inverted file) — partition-based, better for very large collections. DiskANN — disk-based, low-memory serving. Product Quantization (PQ) or Scalar Quantization (SQ) compress vectors.
Quantization. Reduce memory footprint. PQ splits vector into subvectors and quantizes each to a codebook. Trades recall for size.
Sharding. By collection (per-tenant) or by hash. Sharded search fans out queries and merges results.
Serving Layer. gRPC or REST APIs. Client SDKs (Python, JS, Go) with retry and pagination.
Persistence. WAL + snapshot. Recover on restart. Some systems (Pinecone) are managed; you don't see this. Others (Qdrant, Weaviate) let you manage.
End-to-end semantic search flow
Trace a semantic search. User queries "best restaurants in Munich for kids."
Client SDK first embeds the query with the same model used at index time — critical for correctness.
Vector DB receives query with the embedding + metadata filter tenant_id=42 + limit=10.
Search dispatches. Metadata filter is applied first if efficient, otherwise combined with vector search. HNSW graph traversal: start at entry point, greedy hops to nearest neighbors, retreat and explore. Efficient for millions of vectors.
Top 100 candidates returned from graph. Apply metadata filter (if not already). Return top 10 by cosine similarity.
Hybrid variant: parallel BM25 search on text. Reciprocal rank fusion combines top-K from vector and BM25 into final top-10. Higher recall on keyword-heavy queries.
Update pattern: new restaurant added. Client embeds description, calls insert with vector + metadata. Vector DB adds to HNSW graph (incremental) and persistence WAL. Available for search within milliseconds.