Ask a flat vector-search RAG system “who approved the vendor contract that led to the Q3 compliance finding” and it retrieves three or four chunks that each score high on cosine similarity to “vendor contract” and “compliance finding”, then stalls, because the answer isn’t sitting in any single chunk. It’s in the relationship between a person, a contract, an approval event, and a finding that live in three documents written months apart by three different teams. Flat retrieval finds text that resembles the question. It does not find facts that are true about the world and how those facts connect.

A detective’s case board solves a version of this problem that a filing cabinet never could. Pin every name, place, and event to a corkboard and connect them with string, and you can trace “who talked to the courier the night the shipment went missing” by walking string, hop by hop, even though no single case file contains that sentence. A filing cabinet indexed by topic gets you documents that mention couriers and documents that mention shipments. It never gets you the specific path between the two people. That’s the gap between a vector index and a knowledge graph: one finds content, the other finds structure.

Here’s the scale that makes the naive approach fail. Take 2 million source documents (wikis, support tickets, contracts, incident postmortems), extract roughly 8 million raw entity mentions, resolve them down to about 5 million canonical entities connected by 40 million relationship edges, and serve 200 queries/second where a meaningful fraction require 2 hops of reasoning across that graph. A naive fix, embed every chunk and hope semantic similarity approximates multi-hop reasoning, tops out around 60-70% recall on multi-hop questions no matter how good the embedding model is, because similarity is not transitivity. The opposite naive fix, run full graph traversal on every query with no bound, works on a toy graph and falls over the moment one entity (a common vendor, a frequently-cited regulation) has degree in the thousands: your “2-hop” query silently becomes a subgraph with 40,000 candidate nodes and blows your latency budget.

We need to solve for extraction quality at 2-million-document scale without paying frontier-model prices per document, bounded multi-hop traversal that survives high-degree hub nodes, and a fusion layer that combines graph structure with vector similarity into one ranked context without either signal drowning out the other.

Requirements and Constraints

Functional Requirements

• Ingest heterogeneous documents (wikis, CMS pages, support tickets, PDFs) and extract entities and relations from unstructured text
• Resolve duplicate and near-duplicate entity mentions (“Acme Corp”, “Acme Corporation”, “Acme”) to a single canonical node
• Answer multi-hop questions by traversing relationships between entities, not just similarity between text
• Fuse graph traversal candidates with vector search candidates into one ranked context via reciprocal rank fusion
• Attach citations from every claim in the answer back to a source document and span
• Support incremental updates as new documents arrive, without a full graph rebuild

Non-Functional Requirements

• Latency: retrieval budget (traversal + vector search + fusion) at p99 under 400ms; LLM synthesis time-to-first-token (TTFT) around 180ms; end-to-end p99 under 1.2s
• Throughput: 200 req/s sustained query load; extraction backfill throughput sufficient to process 2 million documents inside a 48-hour window
• Quality: recall@k >= 0.85 at k=50 for chunk retrieval; path coverage >= 0.90 on a multi-hop eval set; hallucination rate under 3% on a golden question set
• Cost: retrieval-side cost under $0.05 per 1,000 requests; extraction backfill cost tracked per million tokens processed
• Capacity: graph store memory footprint of roughly 20-30 GB in RAM; vector index of roughly 25-35 GB including HNSW overhead; per-query context budget around 1,200 tokens

Constraints

• Model choices assumed: an open-weight instruction model for bulk extraction (cheap, batched, run self-hosted), a frontier model for final answer synthesis, and an off-the-shelf 1536-dimension embedding model for both chunks and entities
• Hardware assumed: a property graph store shardable across 3+ nodes, a vector database supporting HNSW, and no GPU training infrastructure since this is a retrieval and inference system, not a training pipeline
• Deliberately out of scope: fine-tuning the extraction model, real-time streaming ingestion (that’s its own topic), and full RDF/SPARQL semantic-web tooling, since a property graph covers the traversal patterns this system needs

High-Level Architecture

The system has six major components. The Extraction Pipeline reads raw documents and pulls out entity mentions and relation candidates using a batched LLM pass. The Entity Resolver collapses duplicate mentions into canonical graph nodes. The Graph Store persists nodes and edges and answers bounded traversal queries. The Community Detector runs offline, clustering the graph into topical communities and generating LLM summaries of each one. The Hybrid Retriever is the query-time component that combines graph traversal with vector search. The Answer Synthesizer assembles the fused context into a prompt, calls the LLM, and attaches citations.

A request enters through two paths that run concurrently. On the ingestion side, documents flow through the Extraction Pipeline into the Entity Resolver, which writes canonical entities and edges into the Graph Store and pushes chunk and entity embeddings into the Vector Index. This path runs continuously, decoupled from query serving. On the query side, a user question is embedded and entity-linked at the same time; the Hybrid Retriever fans out to both the Vector Index for semantically similar chunks and the Graph Store for a bounded 2-hop traversal from any entities named in the question, then fuses the two candidate lists with reciprocal rank fusion. The Answer Synthesizer serializes the fused subgraph and chunks into a compact text context, streams a response from the LLM, and maps generated claims back to source document IDs before returning.

The Community Detector sits off the hot path. It runs as a scheduled batch job (daily, or triggered after a large ingestion burst), clustering the graph with the Leiden algorithm and writing a short LLM-generated summary for each cluster back into the graph as a special summary node. Those summaries answer broad questions (“what are the main risk themes across vendor contracts”) that no single 2-hop traversal would surface, because the relevant signal is spread across thousands of nodes rather than concentrated near any one entity.

The Extraction Pipeline

This component turns unstructured text into candidate entities and relations.

The obvious approach, send every chunk to a frontier LLM with a “extract all entities and relationships as JSON” prompt, extracts high-quality triples but costs roughly $12-15 per million input tokens at frontier pricing, which puts a 2-million-document backfill (at ~1,200 tokens/doc average, ~2.4 billion tokens) in the tens of thousands of dollars before a single query is served. The fix is a smaller, self-hosted instruction-tuned model constrained to a strict output schema, run in large batches. Quality drops a little on ambiguous relation types, but for well-defined predicates (works_at, reports_to, approved_by, caused_by) the smaller model gets within a few points of frontier accuracy at roughly 1/40th the cost.