A library card catalog and a reference librarian solve the same problem two different ways. The card catalog matches your exact words against titles and subject headings; it finds “photosynthesis” instantly but misses the shelf on “how plants make food from sunlight” even though that’s the same question. The librarian understands what you mean and walks you to the right section even when your words don’t match any label, but she’s slower and occasionally wrong about which section is right. A retrieval system for a production RAG pipeline has to be both at once, at 200 million chunks and 800 queries per second, without picking one and losing the other.

BM25 is the card catalog. It scores documents by term frequency and inverse document frequency, so an exact product SKU, an error code, or a person’s name will always surface the document that contains it, because no embedding model reliably preserves rare tokens like SKU-88213-B or ECONNRESET in a 1024-dimensional vector. Dense retrieval is the librarian. An embedding model maps “how do I get a refund” and “cancel my order and get my money back” close together in vector space even though they share almost no words, which is exactly what keyword search cannot do. Run only BM25 and you miss every paraphrase. Run only dense retrieval and you miss every exact-match query, and dense retrieval is also known to underperform on out-of-domain terminology it never saw during embedding model training.

The naive fix, take the top 10 from each and concatenate, fails in a specific way: it doubles the context you hand the LLM with irrelevant near-duplicates, blows your token budget before the model reads the real answer, and gives no principled way to break ties when both lists disagree. At 800 req/s with a p99 retrieval budget of 120ms, you cannot afford to run two full searches, merge them badly, and stuff 100 chunks into the prompt hoping the model sorts it out. We need to solve for three things simultaneously: fusing two ranked lists into one without a shared scoring scale, re-ranking that merged list down to the handful of chunks that actually deserve context window space, and doing all of it inside a latency budget that leaves the LLM itself enough time to generate a response.

Requirements and Constraints

Functional Requirements

• Ingest documents from heterogeneous sources (wikis, PDFs, CMS exports, support tickets), chunk them, and keep both a BM25 inverted index and a dense HNSW vector index in sync
• Accept a natural language query and return a ranked, deduplicated set of passages relevant to answering it
• Fuse sparse and dense rankings into a single ordered list using Reciprocal Rank Fusion
• Re-rank the fused candidate set with a cross-encoder before handing chunks to the LLM
• Pack the final passages into a token-budgeted context window, deduplicating near-identical chunks from overlapping windows
• Detect stale source documents and re-embed them without a full index rebuild
• Support per-tenant filtering so a query never surfaces another tenant’s private documents

Non-Functional Requirements

• Latency: end-to-end retrieval (BM25 + dense + fusion + rerank) at p99 under 120ms; p50 under 55ms
• Throughput: 800 queries/second sustained, with headroom to 1,500 during traffic spikes
• Quality: recall@50 above 0.92 on the fused candidate set before reranking; nDCG@10 above 0.78 after reranking
• Cost: under $3.20 per 1,000 queries fully loaded (embedding, search, rerank, LLM generation)
• Capacity: 200 million chunks, roughly 800GB of float16 embeddings at 1024 dimensions, plus a BM25 inverted index around 340GB on disk

Constraints

• Embedding model: a 1024-dimension bi-encoder (bge-large-class), served on GPU, not an API call, because query-time embedding latency has to be single-digit milliseconds
• Reranker: a cross-encoder in the 100-300M parameter range, small enough to score 50 pairs in under 50ms on one GPU
• Hardware: GPU pool for embedding and reranking, CPU-only fleet for BM25 (inverted indexes do not benefit from GPUs)
• Out of scope: multi-hop retrieval, query decomposition into sub-questions, and agentic tool-calling retrieval loops. Those are separate system designs built on top of this one.

High-Level Architecture

The system has six major components. The RAG Gateway receives the query, rewrites it if needed, and fans it out in parallel to both retrieval backends. The BM25 Sparse Index runs inverted-index term matching over the full corpus. The Dense Vector Index runs approximate nearest-neighbor search over an HNSW graph. The RRF Fusion layer merges the two ranked lists into one using reciprocal rank scoring. The Cross-Encoder Reranker takes the fused top 50 and re-scores each query-passage pair jointly, producing the final top 8. The Context Packer deduplicates near-identical passages, trims to the token budget, and hands the assembled context to the LLM.

A request enters at the gateway, which embeds the query with the same bi-encoder used at ingestion time and issues two searches concurrently: a BM25 query against the inverted index and a k-nearest-neighbor search against the HNSW graph, both requesting top_k=50. Both searches return in parallel, typically within 15-25ms of each other, so the gateway waits on the slower of the two rather than paying their latencies serially. The two ranked lists, each containing document IDs and a score that means something different in each list, get passed to the fusion layer.

RRF Fusion ignores the raw scores entirely and works only on rank position, producing one merged list of up to 100 unique candidates ordered by combined rank. That list is too large and too noisy to hand an LLM directly, so the Cross-Encoder Reranker scores the top 50 of those candidates jointly against the query, something neither BM25 nor the bi-encoder can do because both score each document independently of the others. The reranker’s top 8 go to the Context Packer, which drops any passage that is a near-duplicate of a higher-ranked one (common when two overlapping chunks from the same document both pass rerank) and truncates to the token budget before the LLM ever sees the prompt.

The timeline below traces one query end to end with realistic per-stage latency, showing exactly where the 120ms p99 budget gets spent.

The RAG Gateway

The gateway’s job is to take a raw query string, decide what retrieval work needs to happen, and fan it out without becoming the latency bottleneck itself.

A smart engineer’s first instinct is to route: classify the query as “keyword-ish” or “semantic-ish” and send it to only one backend. This is wrong for the same reason a card catalog and a librarian both stay open during business hours: you cannot reliably classify intent from the query text alone, and even queries that look purely semantic (“how do I fix a stuck printer”) often contain a keyword (“printer”) that a document happens to match exactly. Query classification adds latency and a new failure mode (misclassification) in exchange for saving one parallel search that costs almost nothing because it runs concurrently anyway.

The gateway also does light query rewriting: expanding acronyms from a per-tenant glossary, stripping stopwords for the BM25 leg only (the dense leg gets the full query, since embedding models are trained on natural sentences, not keyword soup), and enforcing a maximum query length so a 4,000-word paste doesn’t blow the embedding batch.