A self-storage facility is a good model for what we are about to build. One building, one set of doors, one climate control system, and several thousand customers who each believe they have a private room. The economics only work because the customers share the roof. The business only survives because nobody can open anyone else’s unit, and because the customer who moves a grand piano in at 3am does not keep everyone else from reaching their own things.

That is the vector database problem. We have 40,000 tenants, 2.4 billion 768-dimensional embeddings between them, a 25ms p99 on search, and a recall@10 floor of 0.93. The tenant distribution is the part that decides the architecture: the median tenant holds about 12,000 vectors, the p99 tenant holds 4.1 million, the largest single tenant holds 180 million, and the smallest paying tenant holds 300. The top 40 tenants own 71% of all vectors. The bottom 37,000 own 3.1%.

The naive design is one index per tenant, each in its own pod. It is correct, it is trivially isolated, and it is unaffordable. An HNSW graph needs its resident set in RAM to hit a 25ms p99, and the smallest instance we can rent with meaningful RAM costs about $150 a month. Forty thousand of those is $6 million a month to store 614 GB of index. The 37,000 small tenants would each be paying for 200,000 times more memory than their vectors occupy.

The opposite naive design is one giant index for everybody with a tenant_id filter on the query. That is affordable and it is broken in two separate ways. Filtered HNSW search degrades catastrophically as the filter gets selective: when a tenant owns 0.0001% of the graph, a post-filtered search visits tens of thousands of nodes belonging to other tenants before it finds ten of its own, and recall collapses because the graph’s navigable structure was built for the whole corpus, not for that tenant’s sub-graph. And a single missed filter, anywhere in the stack, is a cross-tenant data leak rather than a bug.

So the real question is not “shared or isolated”. It is: what exactly do tenants share, at which layer, and what is the blast radius when that sharing fails. We need to solve for physical isolation of each tenant’s vectors, latency isolation under a noisy neighbor, and a per-vector cost that works at 300 vectors and at 180 million vectors, simultaneously, on the same control plane.

Requirements and Constraints

Functional Requirements

• upsert(namespace, vectors[]) writes or replaces vectors with metadata, visible to search within 5 seconds
• search(namespace, query_vector, top_k, filter) returns the k nearest neighbors within that namespace only
• delete(namespace, ids[]) and delete_by_filter(namespace, filter) remove vectors, with deletes reflected in results within 5 seconds
• Namespaces are created implicitly on first write and destroyed by an explicit drop_namespace that guarantees the bytes are unreachable
• A single tenant may own many namespaces (typical SaaS pattern: one per end customer), so the system must handle namespace counts an order of magnitude above tenant counts
• Metadata filters support equality, $in, and numeric range on up to 8 indexed fields per namespace
• Per-namespace statistics exposed to the tenant: vector count, dimension, index build state, queries per second, storage bytes

Non-Functional Requirements

Latency. p50 search 6ms, p99 search 25ms, p99.9 search 80ms, measured per namespace and reported per tenant, not as a global aggregate. A global p99 is meaningless here because it is dominated by whichever tenant happens to be large. Cold namespace activation p99 of 250ms, counted separately and budgeted at under 2% of queries.

Throughput. 30,000 aggregate QPS at steady state with a 4x burst ceiling. Sustained ingest of 120,000 vectors per second across all namespaces. A single namespace is guaranteed 200 QPS and may burst to 2,000 QPS when the node has headroom.

Quality. recall@10 of at least 0.93 against exact brute force, measured per size bucket. The pooled small namespaces must not have worse recall than the dedicated large ones, which is the failure mode that filtered-single-index designs hit.

Isolation. Zero cross-namespace result leakage, enforced at three independent layers. A tenant exceeding its quota degrades only its own latency. The 99th percentile latency of a namespace must not move by more than 15% when a co-resident namespace on the same node saturates its quota.

Cost. Under $0.019 per million vectors per hour of resident storage, and under $0.28 per million queries. Cost per tenant must scale sub-linearly with tenant count so the 37,000 small tenants stay profitable at a $30 per month price point.

Capacity. 2.4 billion vectors resident, growing 9% per month. Per-node index RAM of 200 GB usable out of 256 GiB. Local NVMe used as the warm tier for evicted namespaces.

Constraints

• Embeddings are 768-dimensional float32 on ingest, stored as PQ codes plus an NVMe-resident float16 copy for refinement
• Cosine similarity only. We normalise on write and use inner product internally
• Hardware assumed: r6id.8xlarge class nodes, 32 vCPU, 256 GiB RAM, 1.9 TB instance-local NVMe, at about $2.42 per hour on demand
• Out of scope: the embedding model itself, cross-namespace federated search, and geo-replication. Single region, three availability zones
• No GPU in the serving path. We will justify that in the cost section

High-Level Architecture

Six components carry the design.

The Auth and Scope Resolver terminates the request, validates the API key, and resolves the caller to exactly one tenant_id. It is the only place in the system that reads credentials, and the only place that decides which namespaces a request may touch.

The Namespace Catalog is the authoritative map from (tenant_id, namespace) to a placement: a tier, a set of node addresses, and an index generation number. It is a small, boring, heavily cached Postgres table, and it is the component that most teams underbuild.

The Query Router looks up the placement, applies the tenant’s fairness budget, fans out to the owning nodes, and merges results. It is stateless and holds a 30 second cache of the catalog.

The Placement Planner is an offline control loop that decides which tier each namespace belongs to and which node it lives on. It runs every 5 minutes, and it is solving a bin-packing problem with a latency constraint rather than a memory constraint.

The Index Nodes hold the actual HNSW graphs. Each node runs one process managing thousands of independent graphs, one per namespace, with a shared memory allocator and a shared thread pool. This is where the sharing actually happens.

The Warm Tier is instance-local NVMe holding serialised graphs for namespaces that have gone cold, plus the float16 vectors used for exact refinement on every tier.

A search request flows like this. The router receives POST /v1/search with a namespace name in the body and a bearer token in the header. The scope resolver turns the token into a tenant_id, and the router composes the physical key as sha256(tenant_id || namespace). This matters: the tenant identifier comes from the credential, never from the payload, so a tenant cannot address another tenant’s namespace even by guessing its name. The router looks up that key in the catalog cache, gets back a tier and a node list, charges the query against the tenant’s budget, and dispatches.

On the node, the request lands in a per-namespace queue. The engine locates the namespace’s graph in the shared arena, runs HNSW search at an efSearch derived from the namespace’s size rather than a global constant, refines the top candidates against float16 vectors read from NVMe, and returns ids with scores. The router merges shard responses when the namespace is sharded, and then performs one last check: every returned id is verified to belong to the resolved tenant before the response is serialised. That check is cheap and it is the layer that turns a routing bug from a breach into an alert.

The Namespace Catalog