Context and RAG Tooling for Agents

Three libraries, three different bets about what you'll need

All three libraries in this comparison can load documents, chunk them, embed them, and retrieve against a vector store -- the baseline RAG loop is table stakes for each. They differ in what they optimize for beyond that baseline, and that difference determines which one fights you least as the system grows past a prototype.

LlamaIndex: data framework first

LlamaIndex's center of gravity is the data layer: a large catalog of data connectors, index structures beyond flat vector search (tree indexes, keyword-table indexes, knowledge-graph indexes), and query engines that can combine multiple retrieval strategies. It treats "get the right context out of heterogeneous data sources" as the primary hard problem, and agent orchestration as secondary, layered on top of a mature retrieval core.

This makes it the strongest default when the retrieval problem itself is the hard part -- many document types, a corpus that benefits from a non-flat index structure, or a need to combine several retrieval strategies (keyword plus vector plus structured query) for one answer. It's a less natural fit when retrieval is simple (one document type, flat vector search is sufficient) and the actual complexity lives in the agent's control flow instead, where LangGraph or another orchestration-first tool is doing more of the real work.

LangChain: broad integration surface

LangChain's defining trait is breadth: the largest catalog of pre-built integrations across models, vector stores, document loaders, and tools, wired together through a common chaining abstraction (and its LangGraph sibling for more structured orchestration). The pitch is "whatever piece of infrastructure you're already using, there's probably an integration for it already," which matters most for teams whose stack is heterogeneous and changes frequently.

The trade-off that comes with that breadth is real: a library covering this much surface area accumulates abstraction layers, and teams report the sharpest learning curve of the three when they need to step outside the common path and understand what a chain is actually doing underneath. It's the strongest fit when integration breadth is the actual constraint -- swapping vector stores or model providers without a rewrite -- and a weaker fit when the team wants to understand and control every step of a narrow, stable pipeline.

Haystack: production pipeline first

Haystack's center of gravity is the deployable pipeline: components (retrievers, generators, rankers) wired into an explicit, inspectable pipeline graph, with a design history rooted in production search systems rather than notebook prototyping. It tends to feel the most like "software engineering" of the three -- explicit component contracts, less magic, a pipeline definition you can reason about the same way you'd reason about any other data pipeline.

This makes it the strongest fit for a team that already knows its retrieval architecture and wants to build and operate it as a stable production system, with less appetite for the rapid rebinding LangChain's integration breadth optimizes for. It's a weaker fit for early prototyping where the architecture is still being discovered, since its explicitness has an upfront cost that a looser tool doesn't charge.

When to skip a framework entirely

All three add real value when the retrieval problem has genuine variety or complexity worth abstracting over. None of them are free, and for a specific, common shape of system -- one document type, a stable schema, flat vector search, no need to swap vector stores or model providers -- a framework's abstraction layers are pure overhead on top of what a direct embed-and-query call already does in a dozen lines.

The tell is usually retroactive: a team that reaches for a framework "in case requirements change" and never actually exercises the flexibility it paid for in complexity chose wrong. The team that starts hand-rolled and later hits a real need for multiple retrieval strategies, or a rapidly-changing integration surface, and adopts a framework at that point, chose right -- the framework earned its cost against a demonstrated need, not a hypothetical one.

One task, three builds: 10k support tickets

Take a concrete task -- ingest 10,000 historical support tickets and answer questions against them -- and the differences stop being abstract.

In LlamaIndex, the natural path is a VectorStoreIndex built over a SimpleDirectoryReader or a custom data connector for the ticket source, with metadata (ticket status, product area, resolution date) attached per node so retrieval can be filtered, not just similarity-ranked. If tickets vary widely in structure -- some are short one-liners, others long threads -- LlamaIndex's node-parsing layer gives direct control over chunking per document type without leaving the library's own abstractions.

In LangChain, the equivalent is a document loader for the ticket source, a text splitter, an embeddings call, and a vector store wrapper, composed through LCEL (LangChain Expression Language) or a LangGraph node. The work is comparable in size to the LlamaIndex version, but more of it is visibly "glue" -- explicit steps wiring loader to splitter to store -- which is the cost of the integration breadth: nothing is presumed about which pieces you're combining, so every combination is spelled out.

In Haystack, the same task is a Pipeline with named, typed components (a converter, a splitter, an embedder, a writer) connected explicitly by name, and querying is a second pipeline (embedder, retriever, optionally a ranker) rather than an implicit method call. The extra explicitness costs a few more lines up front and buys a pipeline definition that's easy to hand to someone else and have them understand without reading library internals.

Hand-rolled, the same task is: chunk each ticket (a plain function, since the shape is known and stable), call an embeddings API in batches, upsert into pgvector with the ticket metadata as columns, and at query time embed the question and run a similarity query with a WHERE clause for any filters. For this specific shape -- one data source, a schema that isn't changing -- this is typically the shortest path, and every line is something the team wrote and can debug without stepping into a third-party abstraction.

Debugging a bad answer: framework vs. hand-rolled

The comparison that matters more than build time, in practice, is what happens six weeks later when the system returns a wrong answer and someone has to find out why.

With a hand-rolled pipeline, the debugging path is direct: print the retrieved chunks for the failing query, check whether the right ticket was even in the top-k, check the embedding similarity scores, check the chunking boundary didn't split the answer across two chunks. Every step is a function call the team wrote, so there's no question about where to look.

With LlamaIndex or LangChain, the same investigation is possible but goes through the framework's own introspection tools -- LlamaIndex's response objects carry source nodes with scores attached, which is often more convenient than hand-rolled logging once you know to look there; LangChain's tracing integrations (commonly paired with LangSmith) give a similar view but require that tracing be wired up in the first place, which is easy to skip during prototyping and then be missing exactly when a production failure needs investigating. Haystack's explicit pipeline structure means each component's output is a named, inspectable artifact by construction, which tends to make this specific failure mode -- "which stage produced the wrong result" -- the easiest of the three frameworks to localize, because the pipeline graph itself is the debugging map.

The general pattern: framework abstraction that's convenient when the system works is exactly the layer you have to see through when it doesn't. That's not a reason to avoid frameworks, but it is a reason to treat observability (see observability tooling for agents) as part of the initial build, not an add-on for later, regardless of which of the four approaches here is chosen.

What it costs to leave later

A framework chosen for integration breadth or retrieval sophistication is, by construction, harder to leave than a hand-rolled pipeline -- that's the same coin as the value it provides. Migration cost differs meaningfully across the three.

Leaving LlamaIndex usually means re-implementing whatever non-flat index structure was in use (a knowledge-graph index has no simple equivalent outside the library) and re-doing the node-parsing/chunking logic outside its abstractions -- the cost scales with how much of the library's retrieval sophistication was actually load-bearing versus just convenient.

Leaving LangChain is often the cheapest of the three in practice, precisely because of the integration-breadth design: the underlying pieces (a specific vector store client, a specific model provider's SDK) are usually available and usable directly outside the LangChain wrapper, so migration is often "delete the glue, call the underlying SDKs directly" rather than a rewrite from a proprietary format.

Leaving Haystack means re-implementing the pipeline graph's orchestration logic, but because each component's contract was already explicit and typed, translating that graph into hand-rolled function calls is usually mechanical -- the pipeline definition essentially already documents what needs to be rebuilt.

None of this argues against adopting a framework early; it argues for treating "what does leaving cost" as one more input to the choice, alongside the retrieval-sophistication and integration-breadth trade-offs above -- particularly for a team that expects to outgrow whichever choice it makes first.