A library card catalog answers “which book has a chapter on ants” in one lookup. It cannot answer “which country did the scientist who first classified the ant genus that this book covers come from” in one lookup, because that answer does not live on any single card. You have to pull one card, read a name off it, walk to a different drawer, pull a second card, read a date off that one, and walk to a third drawer. Standard RAG is the one-lookup catalog: embed the question, search a vector index once, stuff the top chunks into a prompt, and generate. It works well when the answer sits inside a contiguous span of one document. It fails, quietly and confidently, the moment the answer requires connecting facts that live in different documents written at different times by different people.

The scale that makes this hard: 3 million source documents, 150 complex questions per second, and internal measurement showing the average complex query needs 2.4 sequential retrieval hops to fully resolve, with a long tail requiring 4. A single vector search pass against this corpus tops out around 55% exact-match accuracy on multi-hop questions no matter how good the embedding model is, because the sub-question you’d need to search for (“who is the CEO of the company that acquired X”) does not appear anywhere in the corpus as text. It only exists as a chain of retrievals you have to execute in order.

Naive fixes fail in opposite directions. Retrieve more chunks per pass (top_k=200 instead of top_k=20) and you dilute the context with irrelevant text while still missing facts that require an intermediate lookup the first pass never made. Run the LLM in a raw agentic loop with a generic “search the web” tool and no structure, and you get unbounded tool calls, no defined stopping point, and a latency profile that occasionally blows past 30 seconds because the model decided to take six hops to answer a question that needed two. We need to solve for sub-question decomposition, a bounded and terminating retrieval loop, and evidence aggregation across hops that can be traced back to a citation, all at once, under a 3.5 second p99 budget.

Requirements and Constraints

Functional Requirements

• Accept a natural-language question that may require anywhere from 1 to 4 sequential retrieval hops to answer completely
• Decompose the question into a dependency-ordered set of sub-questions when decomposition is needed, and skip decomposition entirely for questions answerable in one hop
• Bind the answer from an earlier hop into the template of a later sub-question (for example, resolving Who is the CEO of {company}? once company is known from hop 1)
• Retrieve and re-rank evidence independently at each hop using the corpus’s hybrid index
• Aggregate evidence across all hops, deduplicate near-identical chunks, and flag contradictions before synthesis
• Synthesize a final answer with inline citations that trace back to the specific chunk and hop that supported each claim
• Detect and reject circular sub-question dependencies before executing a single retrieval

Non-Functional Requirements

• Latency: p50 end-to-end under 1.8s, p99 under 3.5s, including all hops
• Throughput: 150 complex questions/second sustained, ~360 sub-question retrievals/second at the measured 2.4 hops/query average
• Quality: recall@10 per hop >= 85% against a held-out multi-hop golden set, exact-match accuracy on the final synthesized answer >= 78%
• Cost: under $9 per 1,000 complex questions fully resolved, including decomposition, retrieval, re-ranking, and synthesis
• Capacity: HNSW index for 5 million chunks fitting in under 35 GB of RAM per shard, decomposition model KV cache budget under 10 GB per GPU

Constraints

• Decomposition and confidence scoring run on a small self-hosted model (Qwen2.5-7B-Instruct); final answer synthesis calls a frontier API model
• Hop budget is hard-capped at 4; a question that cannot be resolved in 4 hops returns a partial answer with an explicit “could not fully verify” flag rather than looping further
• Out of scope: web search fallback, multi-turn conversational memory across separate user sessions (covered by a different system), and fine-tuning the decomposition model - it runs as a prompted instruction model

High-Level Architecture

The system has six components. A Decomposition Planner turns the incoming question into either a single retrieval (simple case) or a dependency-ordered sub-question DAG (complex case). A Hop Controller walks that DAG, dispatching one sub-question at a time, tracking a hop budget and a confidence threshold, and deciding after each hop whether to continue or stop. A Hybrid Retriever runs dense HNSW search and sparse BM25 search in parallel for the active sub-question and fuses them with reciprocal rank fusion. A Cross-Encoder Re-ranker takes the fused top-50 candidates down to a precise top-8. An Evidence Aggregator collects the evidence from every hop, deduplicates near-identical chunks, and flags direct contradictions. An Answer Synthesizer takes the aggregated, deduplicated evidence and produces a cited final answer, streamed back to the client over SSE.

A request flows in as a single natural-language question. The Decomposition Planner classifies it and, for anything beyond a single fact lookup, emits a small DAG of sub-questions with explicit dependency edges (sub_q_2 depends_on sub_q_1). The Hop Controller then executes that DAG one node at a time in topological order: for each sub-question it calls the Hybrid Retriever, gets back re-ranked evidence, extracts a bound answer value and a confidence score, and decides whether the hop budget and confidence thresholds allow it to continue to the next hop or whether it should exit the loop early. Once the loop exits, whether by exhausting the DAG or hitting the hop budget, the Evidence Aggregator merges everything collected and hands it to the Answer Synthesizer for the final generation pass.

The Decomposition Planner

This component’s job: turn one question into a validated, dependency-ordered set of sub-questions, or determine that no decomposition is needed at all.