A library that never re-shelves a returned book eventually hands every patron a stale card catalog: the index still points to where the book used to be, or to an edition that has since been revised twice. A RAG system with 50 million indexed documents has the same problem at a much larger scale. Source documents change constantly - a pricing page gets updated, a policy PDF gets a new revision, a wiki article gets edited twenty times a day - and every one of those changes silently invalidates the embedding sitting in the vector index. The model doesn’t know the underlying text moved. It keeps returning the old vector with high cosine similarity, and the LLM confidently answers with information that stopped being true last Tuesday.

The naive fix is to re-embed everything on a schedule: nightly, full reindex, done. That works at 10,000 documents. At 50 million documents, a full reindex means re-running every document through an embedding model, which at 8,000 docs/sec on a 4x A10 GPU pool takes over 90 minutes just for the embedding pass, before you’ve even touched the index rebuild. Running that every night burns GPU-hours you don’t need, because on any given day maybe 4% of documents actually changed. Running it less often means popular, fast-moving documents - the ones most likely to be retrieved - sit stale for days.

The forces in tension are freshness against cost, and index stability against churn. Re-embedding faster costs more GPU-hours and risks flooding the vector index with churn while queries are in flight against it. Re-embedding slower saves money but degrades retrieval quality for exactly the documents users care about most, because a page that changes often is usually a page that matters. On top of that, the embedding model itself will eventually be upgraded, and every one of those 50 million vectors was computed with a specific model version. Mixing vectors from two model versions in the same HNSW graph silently corrupts distance comparisons, because cosine similarity between an old-model vector and a new-model vector is meaningless.

We need to solve for change detection at document granularity without re-reading the whole corpus, incremental re-embedding that scales to a fraction of the corpus per day instead of all of it, and versioned index cutover that lets us swap in a fully re-embedded index atomically when the underlying embedding model itself changes.

Requirements and Constraints

Functional Requirements

• Detect when a source document has changed since its last embedding, at the level of individual documents or chunks, not entire collections.
• Re-embed only the documents that changed, plus any documents whose priority tier demands proactive refresh even without a detected change.
• Support atomic swap-in of newly embedded vectors so no query ever sees a half-updated document (old chunk 1, new chunk 2).
• Support a full corpus re-embed as a distinct, versioned operation when the embedding model itself is upgraded, with zero downtime during the swap.
• Expose a staleness signal per document so downstream consumers (dashboards, SLA reports) can query “how stale is this answer likely to be.”

Non-Functional Requirements

• Latency (freshness SLA): p99 time from source change to updated vector live in the index under 15 minutes for priority=high documents, under 24 hours for priority=low.
• Throughput: sustain re-embedding 2,000,000 documents/day at steady state, with burst capacity to 5x that during a full model migration.
• Quality: recall@10 on a fixed eval set must not regress by more than 1% during any re-embedding rollout; embedding drift (cosine distance between old and new vector for an unchanged document) must stay under 0.02 for same-model re-embeds.
• Cost: target under $0.40 per 1,000 documents re-embedded, inclusive of GPU time, change detection compute, and index write amplification.
• Capacity: change-detection metadata (hash, last-embedded-at, version) for 50 million documents fits in under 40 GB of RAM-resident state; vector index supports holding two full model versions concurrently during migration (2x storage during cutover window only).

Constraints

• Embedding model: a single production embedding model at a time (assume a 1024-dim bi-encoder, similar in size to bge-large or text-embedding-3-large), with model upgrades happening a few times a year, not continuously.
• Hardware: a fixed pool of 4x A10 GPU workers dedicated to embedding, shared with the ingestion pipeline for brand-new documents.
• Out of scope: chunking strategy itself (assume chunks are already produced upstream), the retrieval-time re-ranking stage, and the LLM generation stage.

High-Level Architecture

The system has five major components. The Change Detector watches source systems and computes a content hash per document to decide if anything actually changed. The Staleness Scheduler turns detected changes and priority tiers into a prioritized re-embedding queue. The Re-Embedding Worker Pool pulls from that queue, runs the embedding model, and writes new vectors to a staging area. The Index Version Manager performs the atomic swap of new vectors into the live index, either incrementally per-document or as a full versioned cutover. The Metadata Store persists the hash, version, and staleness state for every document so the whole pipeline is idempotent and resumable after a crash.

A request flows through the system like this. The Change Detector polls or receives a webhook from each source system (CMS, wiki, ticketing system) and computes a sha256 hash of the normalized document body. It compares that hash against the last known hash in the Metadata Store. If the hash differs, the document is enqueued into the Staleness Scheduler with a priority derived from its tier and how long it has been queued. The Re-Embedding Worker Pool dequeues in priority order, batches documents up to the GPU’s optimal batch size, computes new embeddings, and writes them to a staging table keyed by (document_id, embedding_version). The Index Version Manager then either upserts the vector directly into the live HNSW graph (for routine per-document updates) or, during a full model migration, builds a parallel shadow index and swaps a pointer once the shadow index reaches 100% coverage.

The Change Detector

Here’s the job this component does: turn “did this document change” into a cheap comparison instead of an expensive re-embed-and-diff.