Picture a senior librarian who has read every book in a five-million-volume collection. When you ask “where is that thing about leaders in distributed systems?”, she knows you probably want Lamport’s Paxos paper, Raft consensus documentation, or Leslie Lamport’s biography - even though none of those contain your exact phrase. She does not pattern-match your words; she maps your intent to the information space she has memorized.

Keyword search engines are the opposite. They are extraordinarily fast filing clerks who will find every document containing the exact words “leaders in distributed systems” and return zero results if no document uses that exact phrasing. The filing clerk is never wrong - it just answers a different question than the one you asked.

Hybrid ranking combines both: a BM25 sparse retriever (the fast filing clerk) and a dense vector retriever (the semantic librarian) run in parallel, and their results are fused by a scoring function that promotes documents that score well in either dimension. A cross-encoder re-ranker optionally refines the top candidates by scoring each (query, document) pair directly - much more accurate than either retriever alone, but too slow to run over thousands of candidates.

The design challenge is achieving this in under 100ms end-to-end at 5,000 queries per second. That forces every component choice: we cannot afford a slow embedding call before retrieval, cannot afford serial retrieval, and must contain the cross-encoder to at most 20 pairs within a hard 30ms budget gate.

Requirements and Constraints

Functional Requirements

• Accept free-text queries and return ranked documents relevant by both keyword and semantic meaning
• Support 100 million indexed documents averaging 500 tokens each
• Return top-K results (K = 1 to 20) with relevance scores and optional explanations
• Handle zero-keyword-match queries: queries where no indexed document contains the exact query terms
• Support document-level metadata filters combined with semantic search (e.g., “published after 2024, topic = ml”)
• Near-real-time indexing: new documents searchable within 30 seconds of ingestion

Non-Functional Requirements

• Query latency: p50 under 60ms, p99 under 100ms at 5,000 QPS
• Index size: 100M documents; raw text storage separate from search index
• Throughput: 5,000 search queries per second sustained
• Recall@10 for semantic queries: above 85% vs human-labeled ground truth
• Availability: 99.9% for reads; 99.5% for writes
• Embedding freshness: document vectors generated within 30 seconds of document creation

Constraints

• Embedding model is fixed at index creation; model upgrades require full re-embedding
• Cross-encoder re-ranking is gated by a 30ms latency budget: skip if the upstream retrieval is already over 70ms
• BM25 and vector indexes must be co-located per shard to avoid cross-shard latency overhead in the fusion step

High-Level Architecture

The system has five planes: the query processor that encodes the query into both a token stream and a dense embedding, the dual retrieval layer that runs BM25 and ANN search in parallel, the score fusion engine that merges the two ranked lists, the cross-encoder re-ranker that refines top candidates, and the indexing pipeline that keeps both indexes current.

At query time, the Query API Gateway accepts a search request and simultaneously fires two requests: the query text goes to the BM25 inverted index, and the query text goes to the embedding service which converts it to a 768-dimensional dense vector before hitting the HNSW vector index. Both retrievers return their top-100 candidates. The Score Fusion engine merges these 200 candidates using Reciprocal Rank Fusion (RRF), producing a combined ranked list. The top-20 entries from the fused list are passed to the Cross-Encoder service if the latency budget allows; otherwise the fused ranking is returned directly.

Sparse Retrieval: BM25 Deep Dive

BM25 (Best Match 25) is the standard first-stage retriever for keyword-based search. It scores documents based on term frequency (TF), inverse document frequency (IDF), and document length normalization. The key equation for document D and query Q is:

score(D, Q) = sum over terms t of:
  IDF(t) * TF_norm(t, D)

where:
  IDF(t) = log( (N - df(t) + 0.5) / (df(t) + 0.5) + 1 )
  TF_norm(t, D) = tf(t,D) * (k1+1) / (tf(t,D) + k1*(1 - b + b*|D|/avgdl))
  k1 = 1.2, b = 0.75 (tunable)