Why architecture matters here
The reason this is architectural rather than a matter of writing better examples is that the right examples depend on the input, and the input is only known at request time. No amount of prompt-engineering craft can pre-select the ideal demonstrations for a query you have not seen yet. The decision of which examples to show is therefore a runtime computation over a corpus, which is precisely the kind of thing that lives in a retrieval system with an index, a scoring function, and a budget — not in a string constant. Once you accept that, the template becomes a slot the system fills, and the interesting engineering moves to how that slot is filled.
Relevance is the first-order effect, but it is not the only one, and this is where naive nearest-neighbor selection disappoints. If you simply take the top-k most similar exemplars, you frequently retrieve several near-duplicates — the store has many billing examples and they all cluster — so the model sees the same pattern five times and learns nothing about the edges of the task. Effective selection balances relevance against diversity, deliberately spreading the chosen examples across the neighborhood so they collectively map the task rather than redundantly marking one point in it. This is a genuine optimization, not a threshold, and getting it right is most of the value.
Label balance is the second subtle effect. For classification-shaped tasks, if your retrieved examples are all of one class, you bias the model toward that class regardless of the input's true label — the demonstrations quietly become a prior. A selection policy that is aware of labels can enforce a spread, showing both positive and negative cases so the model learns the boundary rather than a majority vote. The same reasoning applies to output length, tone, and format: the examples set expectations along every axis the model attends to, so imbalance on any of them leaks into the answer.
Budget is the constraint that makes all of this a real optimization rather than a wish list. Every exemplar costs tokens, tokens cost latency and money, and long contexts dilute the model's attention on the actual query. You cannot simply add more examples until accuracy stops improving; you must pick the k examples that buy the most behavior per token, which is why the selector is a ranker with a cutoff and not a filter. The architecture has to make the relevance-diversity-budget trade explicit and tunable, because the right point on that curve differs by task and by model.
Finally, freshness turns the exemplar store into a living asset. Because in-context examples carry enormous authority — the model treats them as ground truth for this request — a stale or wrong example is not a mild degradation but an active teacher of bad behavior. The architecture needs a feedback loop that promotes examples which correlate with good outcomes, retires examples that correlate with bad ones, and lets you add new patterns as the task distribution shifts. Without that loop, the store slowly drifts from the world and every retrieval becomes a little more wrong, invisibly, until someone audits it.
The architecture: every piece explained
The exemplar store is the foundation: a curated collection of labeled input-output pairs, each with the input text, the desired output, and metadata such as source, quality score, label, and creation time. This is not a dumping ground for raw logs; it is a curated set where every entry is a demonstration you would be proud to show the model. Each entry is embedded once and stored alongside its vector so retrieval never re-encodes the corpus. The store is the single most important artifact in the system, because retrieval can only ever be as good as what it draws from.
The embedding model encodes both the stored exemplars and the incoming query into the same vector space so that geometric nearness approximates task relevance. The choice of embedding model matters more than teams expect: it must place inputs that call for the same behavior near each other, which is not always the same as placing topically similar text near each other. For many tasks a general-purpose sentence embedder is fine; for specialized domains a fine-tuned or instruction-aware embedder pays for itself by making the neighborhoods actually correspond to task clusters.
The vector index performs approximate nearest-neighbor search so retrieval stays fast as the store grows from hundreds to millions of exemplars. It returns a candidate set — typically more than the final k, because the selector needs room to apply diversity and balance constraints. The index is an operational component with its own latency, memory, and consistency characteristics; it must be updated when exemplars are added or retired, and its recall directly caps the quality of everything downstream, since the selector can only choose from what the index surfaces.
The selector / re-ranker is the brain of the system. It takes the candidate set and applies the real policy: maximize relevance while enforcing diversity (often via a maximal-marginal-relevance style objective that penalizes similarity to already-chosen examples), balance labels, respect the token budget, and honor hard rules such as never showing an exemplar flagged as low-quality. This is where relevance-diversity-budget becomes concrete, and it is the piece most worth investing in, because a mediocre index feeding a good selector beats a great index feeding a naive top-k.
The prompt assembler composes the final prompt: the system instruction, the chosen exemplars formatted consistently (same delimiters, same field order the model will see in the query), and the actual query. Formatting discipline here is not cosmetic — inconsistent example formatting teaches the model that format is negotiable, which is exactly the instability few-shot was supposed to cure. Around all of it sits the feedback loop: outcomes are logged against the exemplars that were shown, quality scores are updated, and curation promotes or retires entries, closing the loop from production behavior back to the store. The diagram shows how these pieces connect for a single request.
End-to-end flow
A request begins when a query arrives at the prompt-construction layer. Before any model call, the system embeds the query using the same embedding model that encoded the store, producing a vector in the shared space. This encode step is on the critical path, so it is typically a fast, possibly local model, and its result may be cached if identical queries recur. The query vector is the search key for everything that follows.
The vector is sent to the index, which returns a candidate set of the nearest exemplars — say the top thirty by cosine similarity. Retrieving more than the final k is deliberate: the selector needs headroom to enforce diversity and label balance, and a wider candidate set makes those constraints satisfiable. The index does the heavy geometric work; the selector does the policy work, and separating them keeps each fast and independently tunable.
The selector now runs its policy over the candidates. It picks the first example greedily by relevance, then iteratively adds examples that are relevant to the query but dissimilar to those already chosen, checks label spread, and stops when the token budget for demonstrations is reached. The output is an ordered short list of exemplars — usually three to eight — that together cover the neighborhood of the query rather than piling onto its single closest point. This is the step that makes dynamic few-shot smarter than plain retrieval.
The assembler formats those exemplars into the prompt template alongside the system instruction and the query, taking care that the examples and the query share the exact same structure so the model reads them as one continuous pattern. The assembled prompt goes to the LLM, which conditions on the demonstrations and produces an answer whose accuracy and format both reflect the chosen examples. From the model's perspective there is nothing dynamic happening — it simply sees a well-chosen few-shot prompt.
After the response, the loop closes. The system logs which exemplars were shown, the produced output, and — where available — a downstream outcome signal: user acceptance, a correctness check, a thumbs rating, or a later human review. Those signals update each exemplar's quality score over time, so demonstrations that consistently precede good answers rise in priority and those that precede failures are demoted or retired. New examples harvested from high-quality production interactions are embedded and added to the store, and the system gradually gets better at teaching itself, one request at a time.