A footnote in a legal brief is worthless if it points to the wrong page. A RAG answer with a citation marker is worthless in exactly the same way if the marker points to a chunk that never said what the sentence claims. That is the problem this system solves: given a generated answer built from 50 retrieved chunks pulled out of 50M documents, map every sentence back to the specific chunk (or chunks) that actually support it, at 200ms or less of added latency, and do it for 500 requests/second without a human in the loop.

Think of it like a research assistant who hands you a report full of confident claims but no footnotes. You cannot ship that to a compliance team, a doctor, or a support agent making a refund decision. You need the assistant to go back through their own draft, sentence by sentence, and write “this came from paragraph 3 of document X” next to each one - and to say “I made this up” when nothing in the source material backs a claim. That second admission is the part most systems skip, and it is the part that actually matters.

The naive approach - just pass the retrieved chunk IDs through and stamp [1][2][3] at the end of the whole answer - fails because it is answer-level, not sentence-level. A three-paragraph response built from 12 chunks tells the reader nothing about which specific claim came from which specific source. The moment the LLM synthesizes information across chunks (which it does constantly - that is the entire value of RAG over plain retrieval), answer-level citation becomes a lie of omission: it looks like attribution but it is really just “here is everything we looked at.”

The forces in tension are real. Sentence-level attribution needs a similarity or entailment check per sentence, which adds latency on the critical path of a response the user is actively streaming. Precision (never citing a chunk that does not support the claim) trades against recall (always finding a citation when one genuinely exists) - tighten the threshold and you get more “unsupported” flags on true claims, loosen it and you get confident-looking citations attached to hallucinations, which is worse than no citation at all. And the system has to do this per-tenant at multi-tenant scale, where a slow attribution path for one heavy customer cannot add tail latency for everyone else.

We need to solve for low added latency per sentence, high precision on the citation-to-chunk match, and graceful degradation to "unsupported" instead of a false citation, simultaneously.

Requirements and Constraints

Functional Requirements

• Accept a generated RAG answer plus the ordered list of chunks used to build the prompt, and return the same answer annotated with citation markers per sentence
• Support streaming: attribution must run incrementally as tokens arrive, not only after the full response is generated
• Attach a confidence score to every citation so downstream UI can render “likely” vs “verified” differently
• Explicitly mark sentences that cannot be attributed to any retrieved chunk as unsupported, rather than silently omitting a citation
• Support multi-chunk citations for sentences that synthesize two or more sources (e.g., “Both the Q2 and Q3 filings show margin compression [2][5]”)
• Expose a query API to fetch the full span map for a given response, for audit and eval tooling

Non-Functional Requirements

• Latency: added attribution latency of <200ms p99 per response, <15ms per sentence on the streaming path
• Throughput: 500 req/s sustained, each response averaging 6-10 sentences needing attribution
• Quality: >=95% attribution accuracy against a human-labeled eval set (precision on cited spans), <3% false-citation rate (citing a chunk that does not actually support the sentence)
• Cost: attribution adds <$0.15 per 1,000 requests on top of generation cost
• Capacity: candidate set per sentence bounded to the k=50 chunks already in the prompt context - never a fresh full-corpus search per sentence

Constraints

• Assumed retrieval stack: hybrid BM25 + dense retrieval already returning chunk_id, document_id, and text for the top-k chunks (see Build a Production RAG Pipeline with Hybrid BM25 + Dense Retrieval)
• Assumed hardware: attribution workers run on CPU (embedding lookup) with a small GPU pool for the cross-encoder reranker, separate from the LLM generation fleet
• Assumed embedding model: the same bi-encoder used for retrieval, so sentence embeddings are directly comparable to chunk embeddings without a second model to maintain
• Out of scope: fact-checking against the world (this system attributes claims to retrieved text, it does not verify the retrieved text is itself correct); citation of information the model already knew before retrieval (parametric knowledge is never cited, only retrieved context)

High-Level Architecture

The system has five major components. The API Gateway authenticates the request and routes it into the pipeline. The Hybrid Retriever runs BM25 and dense retrieval in parallel and fuses results with reciprocal rank fusion, returning the top-k chunks with stable chunk_ids. The Chunk Store is the HNSW-backed vector index plus a metadata table keyed by chunk_id, document_id, and source location. The LLM Generator drafts the answer using the retrieved chunks as context, streaming tokens back over SSE. The Attribution Engine sits between the token stream and the client, buffering tokens into sentences and aligning each one to a source chunk using an embedding filter followed by a cross-encoder score. The Citation Store persists the final span map for audit, eval sampling, and the “why did it say that” support ticket six months from now.

A request flows in as a question, gets embedded and retrieved against the chunk store, and the retrieved chunks (with their chunk_ids preserved) are assembled into the generation prompt with explicit tags so the model can reference them. As the LLM streams tokens, the Attribution Engine buffers until it detects a sentence boundary, then runs that sentence against only the k=50 chunks already in the prompt - never a new corpus-wide search. Each attributed sentence is emitted to the client with its citation markers attached before the next sentence even finishes streaming, and the full span map lands in the Citation Store once the response completes.

The critical design choice is that attribution never re-searches the corpus. It only has to disambiguate among the chunks the generator already saw, which is a k=50 candidate problem, not a 50M document problem, and that is what makes sub-200ms attribution possible at all.