Designing a RAG Pipeline at Scale

RAG is a pipeline, not a single retrieval call

Retrieval-augmented generation gets demoed as one function call: embed the query, search an index, stuff the top results into a prompt. At scale -- millions of source documents, thousands of queries per minute, content that changes daily -- the retrieval call is the last five milliseconds of a pipeline that spans ingestion, chunking, embedding, indexing, and re-ranking, and every one of those upstream stages determines whether the final answer is actually grounded in something true.

This is a systems-design problem with the usual trade-offs: throughput versus freshness, recall versus latency, storage cost versus retrieval quality. This article works through the pipeline stage by stage, with the numbers that make each trade-off concrete.

Ingestion and chunking

Ingestion pulls source documents from wherever they live (object storage, a CMS, a database, a crawl) and normalizes them into text. The design decision that matters most here is chunking: how you split a long document into the units that actually get embedded and retrieved.

Fixed-size chunking (say, 512 tokens with a 10-15% overlap) is cheap and predictable, but it cuts across sentence and section boundaries indiscriminately, which fragments meaning right at chunk edges. Structure-aware chunking (split at headings, paragraphs, or semantic boundaries) preserves meaning better but produces uneven chunk sizes that complicate batching and can blow past a model's context budget if a section runs long. In practice, most production pipelines use a hybrid: split at structural boundaries first, then fall back to fixed-size splitting inside any section that's still too large.

Chunk size is a direct trade-off against retrieval precision. A 2000-token chunk gives the generator more surrounding context per retrieved item but dilutes the embedding -- a chunk about three different subtopics produces a smeared vector that matches all three queries poorly. A 200-token chunk embeds a focused idea well but forces the generator to stitch together many small fragments, increasing the chance a needed piece of context simply wasn't retrieved. 300-800 tokens with 10-15% overlap is the common production range; the right point within it depends on how internally coherent your source documents already are.

Embedding generation and the write path

Embedding is where ingestion throughput actually gets bottlenecked. A single-document, single-request embedding call wastes most of an embedding model's throughput -- production pipelines batch chunks (typically 32-256 per batch, tuned to the embedding provider's rate limits and your own compute if self-hosted) and run ingestion as an asynchronous job queue, not a synchronous step in the document's save path.

document arrives -> chunk -> enqueue chunks for embedding
                                    |
                          embedding worker pool (batched, N chunks/call)
                                    |
                          vector + metadata written to index
                                    |
                          document marked "indexed" (async, eventually consistent)

The write path into the index itself has its own throughput ceiling, and it's a different one than the embedding call's. Most vector indexes (see vector search infrastructure for the index-type trade-offs) are optimized for read (query) throughput, not write throughput -- inserting into an HNSW graph, for instance, gets more expensive as the graph grows, because each insert needs to find its neighbors in an increasingly large structure. High-ingestion-volume systems batch writes and often maintain a small, fast "staging" index for very recent content that gets periodically merged into the main index, rather than writing every chunk into the primary index synchronously.

Retrieval and re-ranking at query time

The query path is the one users feel, so its latency budget is the tightest part of the pipeline. A typical budget: embed the query (10-30ms), search the index for a first-pass candidate set (20-80ms depending on index size and recall target), re-rank that candidate set with a more expensive model (50-150ms), and hand the final top-k to the generator.

The two-stage retrieve-then-rerank pattern exists because the cheap, fast retrieval step (approximate nearest-neighbor search over the whole index) and the accurate, expensive step (a cross-encoder or LLM-based re-ranker that actually reads the query against each candidate) have opposite scaling properties. Retrieval needs to run against millions of vectors and must be fast; re-ranking only needs to run against the top 20-100 candidates retrieval already narrowed down, so it can afford to be five to ten times more expensive per item.

Top-k at the retrieval stage is typically over-provisioned relative to what the generator actually sees -- retrieve 50-100 candidates, re-rank down to the 5-10 that actually go into the prompt. The gap exists because first-pass vector similarity is a coarse signal: the semantically closest chunk by cosine distance isn't always the most useful one for actually answering the question, and re-ranking corrects for that at a cost you can afford exactly because you've already thrown away 90%+ of the index by that point.

Freshness: index staleness versus rebuild cost

Every RAG index is a lagging copy of the source data, and the design question is how much lag is acceptable. A support-documentation index can tolerate hours of staleness; a pricing or inventory index used by a shopping agent cannot tolerate more than minutes, because a stale answer is an actively wrong one, not just an incomplete one.

Incremental indexing (embed and write only what changed, via a change-data-capture stream or a periodic diff against source) keeps staleness low without the cost of touching the whole corpus on every update, but it accumulates its own problem: chunks from deleted or heavily-edited source documents linger in the index as orphaned, misleading entries unless the pipeline explicitly tracks and retracts them. A full rebuild guarantees correctness but costs re-embedding the entire corpus, which at scale is the single most expensive recurring line item in a RAG system's operating cost -- full rebuilds are usually scheduled (nightly or weekly) as a correctness backstop, with incremental updates handling same-day freshness in between.

What actually breaks at scale

Embedding model version drift. Re-embedding only new content with a newer embedding model version than the rest of the index puts two incompatible vector spaces in one index; similarity scores between old and new vectors are meaningless. A model version change requires a full re-embed, not an incremental one -- track the embedding model version as index metadata and refuse to mix.

Chunk-boundary answers. A fact that happens to span a chunk boundary is invisible to retrieval no matter how good the embedding model is, because neither half-chunk contains the whole fact. Overlap mitigates this but doesn't eliminate it; the fix that actually works is structure-aware chunking that respects the semantic units the source content was actually organized around.

Retrieval that returns confidently wrong context. A query with no good match in the corpus still returns the closest vectors, which can be topically related but factually irrelevant -- and a generator handed irrelevant context often still produces a fluent, confident, wrong answer rather than an "I don't have this information" response. A relevance-score floor at the retrieval stage, below which the pipeline declines to answer rather than answering on weak context, is cheap insurance most first versions skip.

Measuring retrieval quality in production

Generator quality is easy to notice when it degrades; retrieval quality degrades silently, because a bad retrieval still produces a fluent answer, just one grounded in the wrong context. Treat retrieval as a component with its own metrics, measured independently of end-to-end answer quality.

The standard metric is recall@k against a golden query set: a curated list of realistic queries paired with the document(s) that should be retrieved for each, run against the live index on a schedule (daily is typical) so an ingestion bug or an embedding-model regression shows up as a metric drop before it shows up as a user complaint. Recall@k alone misses ranking quality within the top-k, so pair it with a ranking metric (mean reciprocal rank or NDCG) that penalizes a correct chunk that's retrieved but buried at position 40 the same way a real re-ranker downstream would be blind to it.

The second production signal worth tracking is the relevance-score distribution of what's actually served, not just the golden-set metric -- a rising share of queries whose top result scores below your relevance floor is an early warning that the corpus has a coverage gap (a new product line, a new integration) the index hasn't caught up to, which a golden set fixed at design time won't catch until someone thinks to add it.