A large enterprise has 10 years of internal documentation scattered across Confluence, Google Drive, Slack archives, and engineering wikis - roughly 2 million documents and growing. An engineer asks the new AI assistant: “What is our incident escalation policy for P0s?” The LLM does not know. It was trained on public data, not your internal runbooks. Without retrieval, it either halts or hallucinates an answer that sounds authoritative but cites procedures from a different company entirely.

Retrieval-Augmented Generation solves this by treating the LLM like a brilliant analyst and your document corpus like the analyst’s reference library. Before the LLM answers, the system fetches the most relevant passages from your knowledge base and hands them to the model as context. Think of it as giving a consultant the exact page of the manual before asking the question - rather than relying on what they remember from reading it three years ago.

The engineering challenge is not just “embed documents and do cosine search.” Enterprise RAG at scale means: documents arrive in 20 formats and need structural understanding before chunking; keyword queries still outperform embeddings for exact-match lookups; embedding models drift over time as internal terminology evolves; re-ranking a candidate pool of 50 documents on every query with a cross-encoder adds 80ms of GPU latency; and measuring whether answers are actually correct requires an evaluation harness that runs continuously.

We need to solve for retrieval quality, query latency, index freshness, and answer correctness simultaneously.

Requirements and Constraints

Functional Requirements

• Ingest documents from Confluence, Google Drive, Slack, and PDF uploads
• Chunk documents preserving semantic coherence
• Embed chunks using a consistent embedding model
• Support hybrid retrieval: dense vector similarity plus keyword BM25
• Re-rank retrieved chunks with a cross-encoder before passing to LLM
• Manage context window assembly to fit within LLM token limits
• Return grounded answers with source citations
• Provide an evaluation framework measuring retrieval and generation quality

Non-Functional Requirements

• Query latency: p95 under 3 seconds including LLM generation (streaming preferred)
• Index freshness: new documents searchable within 5 minutes of ingestion
• Corpus scale: 2 million documents, average 5 pages each, ~10 billion tokens total
• Concurrent queries: 500 QPS at peak
• Embedding throughput: 50,000 chunks/hour during bulk ingestion
• Retrieval accuracy: NDCG@5 above 0.82 on internal eval set
• LLM answer faithfulness: above 0.90 per RAGAS evaluation

Constraints and Assumptions

• Documents require access control - results must respect per-user permissions
• Embedding model is fixed across ingest and query (no mid-flight model changes)
• LLM context window is 128K tokens; we target 4K-8K for retrieved context
• No real-time document editing sync required (5-minute freshness is acceptable)

High-Level Architecture

The pipeline splits cleanly into two paths: the ingestion path that processes and indexes documents, and the query path that retrieves and generates answers.

The ingestion path starts when a document arrives from a source connector. The chunker breaks it into semantically coherent pieces. The embedder converts each chunk to a 1536-dimensional vector. Two indexes receive the output in parallel: the vector store (pgvector or Pinecone) gets the dense embedding, and the BM25 index (Elasticsearch) gets the tokenized text.

The query path starts with a user question. The query encoder converts it to the same vector space as the indexed chunks - using the exact same embedding model ensures the distance calculations are meaningful. The hybrid retrieval layer runs ANN search against the vector store and keyword search against Elasticsearch simultaneously, then fuses the results using Reciprocal Rank Fusion. The top 50 candidates go through the cross-encoder re-ranker, which scores each (query, chunk) pair more precisely. The top 5 land in context assembly, which builds the final prompt and calls the LLM.

Document Chunking Pipeline

The chunker’s job is to split a document into pieces that are small enough to fit in context but large enough to be semantically self-contained. Think of it like cutting newspaper articles to paste into a scrapbook: you want each clipping to make sense on its own, not end mid-sentence.