A library with open stacks is easy to build. A library where every patron has a different set of shelves they’re allowed to browse, where a single book might be visible to the audit team but invisible to the sales team, and where “invisible” has to hold even when the patron asks a very clever question that happens to reference the book’s contents indirectly - that’s a different problem. That is what row-level security in a RAG system actually asks for: not “does this user have an account”, but “for this exact query, which of the 300 million chunks in the index is this exact user allowed to retrieve, right now, before any ranking happens.”

Enterprise RAG systems fail this quietly. A naive implementation embeds every document into one shared index, retrieves the top-k nearest neighbors, and only checks permissions after the LLM has already read the leaked chunk into its context window. By the time a downstream is_authorized() check fires, the model has already seen the finance team’s compensation bands or the legal team’s pending litigation notes. The check is too late because the damage - the token leaving the vector store and entering an LLM’s context - already happened.

At 12,000 users across 500 tenants and roughly 300 million chunks, filtering after retrieval is not just insecure, it is slow. Post-filtering means over-fetching by 5-10x to guarantee enough authorized results survive the filter, which balloons re-ranking cost and blows the latency budget. Doing the ACL check as a first-class part of the vector search itself - a filter pushed down into the HNSW graph walk, not a Python loop after the fact - is what makes this both correct and fast. That single change is the difference between a system that leaks data under adversarial prompting and one that provably cannot.

We need to solve for three things simultaneously: correctness of access control that holds even against retrieval performed by the model on the user’s behalf, latency that stays under 350ms p99 despite the extra filtering work, and scalability across tenants whose document counts range from a few thousand to tens of millions without one noisy tenant degrading everyone else’s recall@k.

Requirements and Constraints

Functional Requirements

• Ingest documents from multiple tenants, each tagged with owning tenant, ACL groups, and sensitivity labels at the chunk level, not just the document level.
• Answer natural-language questions using hybrid retrieval (BM25 + dense vectors) filtered to only the chunks the requesting user’s groups can see.
• Support per-document and per-chunk ACL changes (a document gets reclassified, a user’s group membership changes) with propagation to the index within a bounded staleness window.
• Provide a full audit trail of every chunk considered, retrieved, and passed to the LLM, per query, per user.
• Support tenant-level data residency and deletion (a tenant’s data must be fully purgeable, including from vector shards and caches).

Non-Functional Requirements

• Latency: p99 end-to-end query latency under 350ms including ACL resolution, TTFT under 400ms for the streamed answer.
• Throughput: sustain 800 queries/second across all tenants at peak, with per-tenant fairness so no tenant can starve another.
• Quality: recall@10 for the ACL-filtered result set must not regress by more than 3% relative to an unfiltered baseline - the filter should exclude unauthorized chunks, not accidentally exclude authorized ones.
• Cost: retrieval and re-ranking cost per 1,000 requests under $4.50 at the target scale.
• Capacity: 300 million chunks total, up to 50 million chunks for a single large tenant, ACL bitmap index sized to fit in RAM per shard (target under 12GB per replica).

Constraints

• Assume embeddings come from a 1024-dimension bi-encoder (text-embedding-3-large-class model) and re-ranking from a cross-encoder served on GPU.
• Assume the identity provider (Okta/Azure AD-class IdP) is the source of truth for group membership; the RAG system caches resolved groups but never invents them.
• Out of scope: field-level redaction inside a single chunk (we assume ACLs are chunk-granular, not sub-chunk-granular), and cross-tenant federated search (each query is scoped to exactly one tenant).

High-Level Architecture

The system has six components. The AuthZ Gateway sits in front of everything and resolves the caller’s identity into a concrete set of group IDs before a single vector is touched. The Retrieval Layer runs BM25 and dense search in parallel, both scoped by the resolved ACL groups, and fuses the results with reciprocal rank fusion. The GPU Pool hosts the cross-encoder re-ranker and the generation model. The Vector Index is sharded by tenant and stores an ACL bitmap alongside every chunk’s embedding. The Policy Store is the system of record for which groups can see which documents, kept in sync with the IdP. The Audit Log records, for every query, exactly which chunks were considered and which were filtered out and why.

A request enters through the gateway carrying a bearer token. The gateway resolves that token into {user_id, tenant_id, group_ids} and attaches an ACL bitmap (a roaring bitmap OR of the group grants) to the request context. That bitmap travels with the query into the retrieval layer, where it is pushed into both the HNSW graph walk and the BM25 posting-list scan as a hard filter, not a post-hoc one. Only chunks whose stored ACL tag intersects the caller’s bitmap are ever scored. The fused, filtered candidates go to the cross-encoder for re-ranking, the top 8 go to the LLM, and every chunk that was considered - kept or dropped - is written to the audit log asynchronously so the write never sits on the request’s critical path.