Vector Search Infrastructure

The index-type decision most teams get to skip, until they can't

Vector databases covers the database-level mechanics of storing and querying embeddings. This article is one layer up: the infrastructure and capacity-planning decisions that only surface once a vector index stops being "a table with a vector column" and starts being a distributed system in its own right -- which index algorithm to run, how to shard it across machines, and how to reason about the recall-latency-memory triangle every approximate index sits inside.

Most teams never need this layer -- a single-node index handles tens of millions of vectors comfortably. This article is for the point past that, where the shape of the data or the query volume forces real infrastructure decisions.

Index types and what they actually trade off

Flat (brute-force) search compares a query vector against every stored vector directly. It's exact -- 100% recall by construction -- and for a few hundred thousand vectors on modern hardware it's fast enough that there's no reason to reach for anything approximate. Past that scale, comparing against every vector for every query stops being affordable, and that's the point approximate methods exist to solve.

HNSW (hierarchical navigable small world) builds a multi-layer graph where each vector is linked to its approximate nearest neighbors; a query walks the graph from a coarse top layer down to a fine bottom layer, following the closest-looking edges at each step. It gives very good recall at very good query latency, at the cost of a larger memory footprint (the graph structure itself, on top of the vectors) and slower, more expensive inserts as the graph grows -- which is why the write path discussion in RAG pipeline design treats high-ingestion-volume systems as needing a separate staging strategy rather than writing straight into a large HNSW index.

IVF (inverted file index) clusters the vector space into a fixed number of partitions (via k-means or similar) at build time, and a query only searches the partitions closest to it rather than the whole index. It uses less memory than HNSW for the same corpus and inserts more cheaply, but recall is more sensitive to how well the partition count and search-time partition count (nprobe) are tuned to the actual data distribution, and a corpus whose distribution shifts after the index is built (partitions no longer reflect where the data actually clusters) needs periodic rebuilding to keep recall up.

Sharding an index across nodes

Past the point a single node can't hold the index in memory (or can't serve query volume fast enough alone), the index has to shard. Two shard strategies dominate, and they trade off differently.

Horizontal sharding with full fan-out is the default because it's operationally simple and works correctly regardless of data distribution; it's usually the right starting point. Semantic sharding earns its complexity once fan-out cost (every shard doing work on every query, even ones with no relevant vectors) becomes the actual bottleneck -- typically well past a dozen shards.

The recall/latency/memory triangle

Every approximate index exposes tuning knobs that trade these three against each other, and the trade is real, not a bug to engineer away: HNSW's ef_search (how many candidates to explore per query) and M (how many edges per graph node), IVF's nprobe (how many partitions to search) and partition count -- all of them buy recall at the cost of either latency (explore more) or memory (store more structure).

The practical approach is to fix a target recall (measured against a held-out ground-truth set, the same discipline as the golden-query-set evaluation in RAG pipeline design) and tune the cheapest knob that hits it, rather than maximizing recall unconditionally -- 99% recall at triple the latency of 95% recall is rarely worth it for a downstream re-ranking step that's about to throw away 90% of the candidates anyway.

Filtered vector search

Real queries are rarely pure semantic search -- "find similar documents, but only ones this user has access to" or "only from the last 30 days" is the common shape, and how filtering combines with approximate search matters more than it looks.

Post-filtering (run the vector search first, then discard results that fail the filter) is simple but breaks down when the filter is selective: if only 2% of the corpus matches the filter, a top-100 vector search result set might contain zero filter-passing items, and the query returns nothing despite matches existing in the index.

Pre-filtering (restrict the candidate set to filter-passing vectors before or during the search) avoids that failure but is harder to make fast, because it doesn't compose cleanly with graph-based indexes like HNSW -- the graph's navigability assumes it can walk toward any neighbor, and restricting that mid-walk to only filter-passing nodes can break the approximation's guarantees. Indexes built with filtering as a first-class concern (rather than bolted on) handle this by incorporating filter predicates into the index structure itself, at some cost to build complexity and memory.

When pgvector is enough, and when it isn't

An extension on a database you already operate (pgvector on Postgres is the common case) has one overwhelming advantage: no new operational surface. No new system to deploy, monitor, back up, and staff expertise for -- vectors live next to the relational data they're already joined against, in transactions that already work the way your team understands transactions to work.

It's enough for most applications: pgvector's HNSW and IVFFlat support handle corpora into the tens of millions of vectors with entirely reasonable latency on hardware a normal Postgres instance already runs on. The point it stops being enough is usually one of: query volume that needs horizontal scaling Postgres itself doesn't give you for free, a corpus large enough that the index no longer fits comfortably in memory alongside everything else Postgres is doing, or a need for vector-index-specific operational features (live index rebuilding without downtime, fine-grained recall/latency tuning per collection) that a purpose-built vector database exposes and a general-purpose database's extension doesn't.

Reach for a dedicated system (self-hosted or managed) at that point, not before -- the operational cost of running a second stateful system is real, and "we might need to scale eventually" is not, by itself, a reason to pay it today.

Operational failure modes specific to this layer

Index rebuilds that block writes or reads. Some index structures can't be updated incrementally in place for certain operations (changing distance metric, changing key tuning parameters like HNSW's M) and require a full rebuild -- if that rebuild isn't designed to happen alongside the live index (build a new index in the background, then atomically swap), it becomes a maintenance window, which is a hard sell for anything user-facing. Plan for dual-index rebuilds (old serves reads while new builds, then swap) from the start rather than retrofitting it under pressure the first time a rebuild is needed.

Memory pressure from graph-based indexes creeping past provisioned capacity. HNSW's memory footprint isn't just the vectors -- it's the vectors plus the graph edges plus per-node bookkeeping, and it grows faster than raw vector count alone would suggest as the corpus grows, because higher M (more edges per node, needed to hold recall steady as the corpus gets larger and more crowded) multiplies the graph overhead. Capacity planning that budgets memory off vector count and dimensionality alone, without accounting for graph overhead at the corpus's target scale, under-provisions in a way that doesn't show up until the corpus actually reaches that scale.

Embedding dimensionality changes forcing a full re-index. Switching embedding models -- even to a "better" one -- almost always means a different vector dimensionality and a different vector space, which means every existing vector in the index is now incompatible with new queries embedded by the new model, the same version-drift problem covered for RAG pipelines. There is no incremental path across an embedding model change; budget a full re-embed and re-index as part of any embedding model upgrade, not as an afterthought discovered mid-migration.