Permission-Aware RAG: Filtering by Access Control Without Destroying Recall

The rule: enforce access in retrieval, not in the model

Access control for RAG has one non-negotiable property: a passage the user may not read must never be placed in the prompt, cached for another user, or written to a trace that the wrong people can open. Everything after retrieval is too late. Language models do not keep secrets reliably, prompt injection can ask them to reveal context, and "please do not mention restricted documents" is an instruction, not a boundary.

In practice this means each indexed chunk carries the identifiers of the principals (users, groups, roles) allowed to read its source document, copied from the source system at ingestion time, and every query carries the requesting user's principals, resolved by your identity provider when the request arrives. The vector search must return only chunks whose principal list intersects the user's. The rest of this guide is about how to do that without wrecking retrieval quality, and how to keep the copied ACLs honest.

Vectors deserve the same protection as text. Embedding inversion research (Morris and colleagues, 2023) showed that a large share of the original text can be reconstructed from some embeddings, so an index of restricted documents is restricted data, including its backups and any debugging dumps.

Selectivity: the number that decides everything

Selectivity is the fraction of the corpus a given user may see. In a typical organisation it spans two orders of magnitude: an administrator or a manager sees most documents, a team member sees their team's spaces plus shared material, and a contractor sees one project. The arithmetic of every filtering strategy depends on it.

Post-filtering, the simplest approach, runs an ordinary nearest-neighbour search for k' candidates and drops the ones the user cannot see. If allowed documents were scattered at random, the number that survive would be roughly binomial with mean k' x s:

survivors ~ Binomial(k', s), expected = k' x s

want 10 results
s = 0.60 (broad user)          : k' ~ 17 is enough on average, ~25 to be safe
s = 0.10 (team user)           : k' ~ 100, ~150 to be safe
s = 0.02 (single-group user)   : k' ~ 500, ~700 to be safe

k' = 10 for everyone: broad gets ~6 results, team ~1, single-group ~0.2

Two facts soften and sharpen this. Softening: allowed documents are rarely random relative to the query. A team's documents are about the team's subjects, and the team mostly asks about those subjects, so the local selectivity near the query is far higher than the global one. Sharpening: when a user asks about something outside their area, which happens constantly in cross-functional work, the local selectivity can be close to zero, and no reasonable oversampling factor saves post-filtering. In the lab, the cross-team event shows exactly this collapse for single-group users.

Filtered graph search and why it slows down

The second approach pushes the filter into the approximate search itself. Graph indexes such as HNSW navigate by moving between neighbouring nodes; a filtered search still walks through all nodes, but only admits allowed ones into the result set. hnswlib exposes this as a filter callback, Elasticsearch applies a filter inside the knn clause during the graph search, and Qdrant combines its HNSW graph with payload indexes.

The catch is the stopping rule. A beam search with width ef stops when the closest unexplored candidate is further than the worst member of a full result set. If allowed nodes are rare, the result set takes a long time to fill, so the search keeps expanding, sometimes through most of the graph. Recall stays good because the search eventually finds the allowed nodes, but the number of distance computations grows towards a full traversal, and with random memory access a full traversal of a large graph is far slower than a sequential scan of the same data. If the implementation instead refuses to traverse through disallowed nodes, the graph restricted to allowed nodes can fall apart into disconnected pieces and recall collapses rather than latency.

Research systems attack this directly. ACORN builds a denser graph so that the subgraph induced by any predicate stays navigable, and Qdrant's filterable HNSW adds extra links for values of indexed payload fields so that common filters keep the graph connected. Both trade index size and build time for predictable filtered search. For ACL filters with thousands of distinct groups, check whether such per-value links are feasible at your cardinality.

Pre-filtering with exact scans, and the planner

The third approach inverts the order: find the allowed set first, through an ordinary inverted index on the principal field, then compute exact distances to every allowed vector. Recall is perfect by construction. Cost is proportional to the allowed set: cheap for a contractor who sees 2% of a 5 million chunk corpus (100,000 vectors, a few milliseconds with SIMD), expensive for a manager who sees 60% (3 million vectors, over a hundred milliseconds in the lab's illustrative cost model).

The two costs cross, which is why production engines use a planner: estimate the size of the allowed set from the payload index, scan exactly when it is small, and run filtered graph search when it is large. Qdrant documents this behaviour with a full-scan threshold in its HNSW configuration; check your version's documentation for the units and the default, which are easy to misread. The decision rule is simple to write and needs tuning per corpus:

def search(query_vec, principals, k=10):
    allowed = acl_index.estimate_count(principals)     # cardinality from the payload index
    if allowed < SCAN_THRESHOLD:                        # e.g. a few percent of the corpus
        ids = acl_index.lookup(principals)               # exact candidate set
        return exact_top_k(query_vec, ids, k)            # perfect recall, cost ~ len(ids)
    return hnsw.search(query_vec, k, ef=EF, filter=principals)  # filtered graph walk

Set the threshold too low and small-but-not-tiny allowed sets go through the graph walk that never fills its result set; set it too high and large allowed sets go through multi-million-vector scans. Mission four of the lab asks you to find the band between those two failure modes for three user classes at once.

The knobs in real systems

Semantics differ in ways that matter: whether a filter is applied before, during or after the approximate search decides whether you get k results or fewer, and whether the engine silently falls back to brute force decides your latency. Read the documentation for the exact version you run and test with your real selectivity distribution, not a single test user.

When access boundaries coincide with tenants (separate customers, separate legal entities), partitioning is cleaner than filtering: one index, collection, namespace or partition per tenant gives perfect isolation and per-tenant recall equal to the unfiltered case. Partitioning does not work for overlapping groups inside one organisation, where a document can belong to many groups and a user to many more; that is where the filtering strategies above are unavoidable.

Permission changes: the leak nobody tests

The ACL in the index is a copy. When someone loses access in the source system (a person changes teams, a project space is locked down, a share link is revoked) the copy stays wrong until the next sync. With a nightly sync, every revocation leaks for up to a day. The leak is quantifiable: if half your users lost one group each, the lab's model returns hundreds of revoked documents per thousand queries until the index catches up.

Three defences combine well:

1. Event-driven ACL sync. Subscribe to permission-change events from the source systems and update the principal field of affected chunks within seconds. Updating a payload field is cheap compared with re-embedding, and the hard part is reliable change capture, not the index write.
2. Late-binding checks. Before passages enter the prompt, verify the final handful against the live permission service. It is the strongest guarantee and costs a few milliseconds for a batched lookup, but it removes results after ranking, so the user may receive fewer than k passages unless you overfetch.
3. Permission-aware caches. Any cache of retrieval results, reranked lists or generated answers must be keyed by the user's principal set (or a hash of it) and invalidated on permission change. A semantic cache keyed only by the query text is a cross-user leak waiting for two people to ask the same question.

Resolve the user's principals at query time from the identity provider rather than from a long-lived session claim, and log which principal set each request used, so that an audit can show why a passage was visible.

A worked example: the Belgavi's Book Store staff assistant

Suppose the hypothetical Belgavi's Book Store runs a staff assistant over 5 million chunks from policies, supplier contracts, store reports and team wikis. Store managers see about 60% of the content, department staff about 10%, and seasonal contractors one project space, about 2%. In the lab's illustrative cost model, a post-filter with 10 candidates gives managers about 75% recall but contractors about a third, because most of the 10 nearest chunks are not theirs. Raising the candidate count to 200 lifts the worst class above 90% at around 8 ms, as long as people ask about their own area.

Then the contractors start asking about store-wide policies, which mostly live in spaces they cannot see. Post-filter recall for them drops below 35% even at 300 candidates. A pre-filtered exact scan fixes recall for everyone but costs managers a 3-million-vector scan, about 130 ms at p95. A planner that scans when the user sees under about 5% of the corpus and uses filtered graph search otherwise gives near-perfect recall for all three classes at roughly 15 ms p95. Finally, a reorganisation revokes access for half the staff; with nightly sync the assistant would serve revoked documents until morning, so the team moves ACL updates to change events and keeps a live check on the final passages. These are calculated estimates from a synthetic model, but the sequence of decisions is the one real deployments go through.

Detecting problems in production

Permission bugs are invisible in aggregate metrics because the users most affected are the ones with the least access, and they are a minority. Instrument by selectivity class:

• Record the allowed-set size estimate per query, bucket it, and report recall on a labelled set, results returned, and latency per bucket.
• Alert on the share of queries returning fewer than k results; a rise usually means post-filter starvation or an over-aggressive late check.
• Track ACL lag: the time between a permission change event and the index update, as a percentile, and page on it like any other data freshness SLO.
• Run a standing canary: a test principal that is granted and then revoked access to a canary document on a schedule, with a probe that fails if the document is retrievable after revocation.

The common mistakes: testing with an administrator account that sees everything, enforcing permissions in the prompt or in the application after generation, caching across principals, syncing ACLs less often than content, and tuning oversampling on average users while contractors receive empty contexts.

How the lab maps to production

The lab builds a synthetic index of 1,600 chunks in 20 topics, assigns each to one of 50 groups (mostly aligned with topics), and builds a neighbour graph for HNSW-style beam search. Synthetic users belong to 30, 5 or 1 groups and ask questions near their own groups' topics, or anywhere with the cross-team event. Each strategy runs for real on that index: post-filtering, filtered beam search with hnswlib-style stopping, exact scans of the allowed set, and a planner. Latency is extrapolated to a 5 million chunk index with stated assumptions. Mission one asks you to make post-filtering serve single-group users. Mission two breaks post-filtering with cross-team questions. Mission three revokes access and asks you to stop the leak without losing recall. Mission four asks you to tune the planner so every user class meets the same recall and latency targets.

Run the lab: Filter by ACL without starving users or leaking revoked documents

The lab below serves three kinds of synthetic users from one index. The left panel is a 2-D view of the corpus for one query at a time: lit dots are chunks the current user may see, green dots are true top-10 results that were found, red rings are ones that were missed, and amber marks revoked documents still visible in the index. The right panel shows recall@10 for each user class. Start with the naive post-filter, then load mission one.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Compute the selectivity distribution of your users (fraction of chunks each can see) and test retrieval for the lowest-selectivity group, not an admin account.
2. Confirm whether your engine applies ACL filters before, during or after the approximate search, and how many results it returns when the filter is selective.
3. Configure a planner threshold (or equivalent) so small allowed sets are scanned exactly and large ones use filtered graph search; verify with per-bucket latency.
4. Move ACL updates to permission-change events and track the lag between a change and the index update as an SLO.
5. Add a batched live-permission check on the final passages before they enter the prompt, and overfetch slightly to keep k results.
6. Key every retrieval and answer cache by the user's principal set, and add a revocation canary that fails if a revoked document is still retrievable.

Sources and further reading

• ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data (SIGMOD 2024)
• Qdrant documentation: Indexing (payload index and filterable HNSW)
• pgvector README (filtering and iterative index scans)
• Elasticsearch reference: k-nearest neighbor (kNN) search
• hnswlib: Header-only C++/Python library for fast approximate nearest neighbors
• Text Embeddings Reveal (Almost) As Much As Text (EMNLP 2023)

Related in this series:

• Keeping RAG Indexes Fresh: Incremental Updates, Deletes and Tombstones
• HNSW Tuning for RAG: M, efSearch and the Recall Versus Latency Trade
• Semantic Caching for LLMs: Similarity Thresholds and the Wrong-Answer Risk

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.