Memory and Vector Store Options for Agentic Systems

The real question is operational, not algorithmic

Every option on this list can compute approximate nearest-neighbor search over embeddings -- the algorithmic core is not where these systems actually differ for most teams. Where they differ is what it costs to run one, how it fits into a system that already has a database, and what happens at the query volume and dimensionality where the easy option stops being easy. This site's vector database deep-dive covers a single technology's internals; this article is the decision between technologies, for the specific case of an agent's memory layer -- conversation history, retrieved facts, and long-term episodic recall.

Three shapes cover almost every real deployment: a vector extension on a database you already run, a managed hosted vector database, and a self-hosted open-source vector database. Each is a genuinely different operational bet, not just a different API.

pgvector: one database, one operational surface

pgvector adds vector columns, distance operators, and approximate-index types (IVFFlat, HNSW) directly to Postgres. The pitch is almost entirely operational: if the rest of the application's data already lives in Postgres, agent memory can live in the same database, transactionally consistent with everything else, backed up by the same process, monitored by the same dashboards, and queried by the same connection pool.

The practical ceiling is real but higher than its reputation suggests: HNSW indexes in modern Postgres handle millions of vectors with sub-100ms query latency on reasonable hardware, which covers the memory store for the overwhelming majority of single-tenant or moderate-multi-tenant agent deployments. Where it genuinely strains: very high query throughput contending with the same database's transactional write load, vector counts in the tens of millions and beyond where index build/rebuild time becomes an operational event, and workloads wanting index types or filtering performance pgvector doesn't yet match a purpose-built engine on.

The failure mode to watch for is not capacity -- it's coupling. A vector workload that grows faster than the rest of the schema starts dominating the shared database's resource budget, and now a memory-layer scaling problem is also a core-application-availability problem. That coupling is the actual argument for splitting out, not raw vector count.

Managed vector databases: pay for not operating it

A managed, hosted vector database (Pinecone is the reference example of this shape) sells the elimination of the operational surface entirely: no index tuning, no capacity planning, no upgrade windows, a control plane and an API. For a team without dedicated infrastructure engineering, or one that wants to spend its engineering time on the agent's behavior rather than on database operations, this is a legitimate and often correct trade -- the cost is a recurring bill and a new external dependency in the critical path, not a comparable amount of engineer-hours.

The trade-offs worth pricing in explicitly: data now lives outside your own infrastructure boundary, which has real implications for data residency and compliance depending on what's in the embeddings' source text; latency includes a network hop to a third-party service rather than a local database round-trip; and the bill scales with usage in a way that's easy to underestimate at prototype scale and genuinely expensive at high query volume. None of these are disqualifying -- they're the actual price of the operational simplicity, and for many teams that price is worth paying.

Self-hosted open-source vector databases: control without the managed premium

A self-hosted open-source vector database (Qdrant is the reference example) sits between the two: purpose-built for vector search specifically, so it typically outperforms a general-purpose database's vector extension at scale and offers filtering/payload features tuned for retrieval workloads -- but you're running and operating it yourself, which reintroduces the operational surface pgvector's pitch eliminates and the managed option's pitch also eliminates.

This is the right choice when a team has both the infrastructure capability to run a new stateful service well (backups, upgrades, capacity planning, on-call coverage) and a genuine reason to avoid the managed option's cost or data-locality constraints. It's the wrong choice when neither is true -- a team that reaches for a dedicated vector database purely because it sounds more "production-grade" than an extension on their existing Postgres, without the operational capacity to run it properly, often ends up with a worse-operated system than either alternative.

The decision point, stated as a checklist

Start with pgvector if there's already a Postgres in the stack -- it's the only option with zero new operational surface, and its ceiling is higher than most agent memory workloads will ever reach. Move off it only when a specific, measured constraint forces the move: sustained query latency degradation under real load, index rebuild time becoming an operational problem, or write contention with the primary application's transactional workload. Choosing a dedicated vector system before hitting one of those constraints is optimizing for a scale the workload hasn't reached yet, at the cost of operational complexity it's already paying for.

Hybrid: one database or two

Agent memory is rarely pure vector data -- there's usually structured state alongside it (conversation metadata, user records, session state, tool-call history) that a vector-only engine handles poorly or not at all. That makes the real architectural question, for most systems, not "which vector store" in isolation but "one database holding both, or a split between a structured store and a vector store."

Keeping both in Postgres (structured tables plus pgvector columns) means every query that needs both -- "find the three most similar past conversations for this user, filtered to the last 30 days, joined against their account tier" -- is a single SQL query with a normal join, transactionally consistent with the rest of the write path. That consistency is easy to underrate until it's gone: splitting structured state into one system and embeddings into another means every cross-cutting query becomes an application-level join across two round-trips, and every write that touches both needs its own consistency handling (what happens if the structured write succeeds and the vector write fails?) that Postgres gives you for free inside one transaction.

The case for splitting anyway is the same case as the rest of this article: a specific, measured need the combined system can't meet -- vector query latency or throughput that's degrading because it's contending with the structured workload on the same hardware, or a filtering/index capability only a dedicated engine offers. Absent that specific pressure, the default that avoids an entire class of cross-system consistency bugs is keeping both together.

Estimating when you'll actually outgrow pgvector

"Millions of vectors" is not a useful planning number on its own -- the three variables that actually determine whether pgvector holds are query throughput, vector dimensionality, and latency tolerance, and they trade off against each other.

A concrete reference point: an HNSW index over roughly 5 million vectors at 1536 dimensions (a common embedding size), on a database instance with a few dedicated CPU cores and enough RAM to keep the index resident, typically holds sub-100ms query latency at a few hundred queries per second before contention with concurrent writes becomes visible. Halving the dimensionality (768, also common) roughly doubles the vector count achievable at the same latency and throughput target, since index size and comparison cost both scale with dimensionality. Conversely, a write-heavy workload -- frequent inserts alongside reads, as with a fast-growing conversation history -- degrades the achievable read latency at a given vector count faster than a read-mostly workload, because HNSW index maintenance under concurrent writes has its own cost that a purpose-built vector engine's write path is often built to isolate more cleanly.

The practical estimation exercise: take the actual expected vector count at 12 months (not today's count), the actual embedding dimensionality already chosen, and the actual required p99 latency, and load-test pgvector at that shape before deciding anything -- not as a one-time gate, but because the honest answer is workload-specific enough that the general reference numbers above are a starting point for a test, not a substitute for one.

Migrating off pgvector later

Because pgvector is a small set of column types, index definitions, and query operators layered onto an existing schema, migrating off it later is more contained than migrating off most other infrastructure choices -- but it's not free, and knowing the shape of the work in advance is part of the decision to start there.

The work breaks into three pieces. First, the data itself: embeddings and their associated metadata need to be read out of Postgres and written into the new system, which for a large table is a batched export/import job, not a single migration script -- budget for it to take real wall-clock time proportional to table size, run against a replica if the primary can't absorb the read load. Second, the query layer: every application code path that issued a SQL query against the vector columns needs to be rewritten against the new engine's client/API, which is mechanical but touches every call site, not just a config change. Third, and easiest to underestimate: the structured-data queries that used to be a single joined SQL statement against the combined table (the hybrid case above) now need to be recomposed as two queries against two systems plus application-level joining, which is exactly the complexity the hybrid section described avoiding -- and is the actual bulk of the migration effort for a system that had leaned on that join pattern.

None of this argues against migrating when the load-test above says to. It argues for treating "what does leaving cost" as part of the initial decision, the same way the RAG-tooling article treats framework migration cost -- pgvector's low switching cost, relative to a bespoke schema built entirely around a different engine's data model, is itself one more point in favor of starting there by default.