A corporate mailroom that receives ten thousand envelopes a day doesn’t hire one clerk to open, read, and route every piece by hand. It runs an assembly line: a machine slits envelopes, a scanner reads the address, a sorter routes by department, and only then does anything reach a desk. Do it any other way and the backlog outgrows one clerk’s throughput before lunch. A chunking and embedding pipeline for 100 million documents is the same problem at a much larger scale: raw text arrives faster than any single process can turn it into something a vector index can search, and the naive approach, one loop that reads a document, splits it into passages, embeds each one, and writes it to the index synchronously, collapses long before it reaches document ten million.

The corpus here is 100 million documents (wikis, PDFs, CMS exports, support tickets) averaging 1,800 tokens each. Chunked into 512-token windows with 15% overlap, that becomes roughly 420 million chunks, each needing a 1024-dimension embedding before it’s searchable. The backlog has to clear in 5 days, which works out to ~1,000 embeddings/second sustained, and once the backfill is done, a 2.5 million document/day stream of edits and new content has to stay fresh enough that an edited document is searchable again within 15 minutes. Neither number is the hard part alone. The hard part is that both have to be true on the same pipeline without provisioning for peak backfill load forever.

The naive synchronous loop fails in a specific way: it couples I/O-bound work (reading from a CMS API, writing to Postgres) with GPU-bound work (the embedding forward pass) in the same execution path, so the GPU sits idle waiting on network calls and the network sits idle waiting on the GPU. Worse, re-running the loop after a crash re-embeds documents it already finished, because nothing tracked what succeeded. At 100 million documents that means wasted GPU-hours measured in thousands of dollars, not a rounding error. We need to solve for three things simultaneously: decoupling ingestion, chunking, and embedding into independently scalable stages; making every stage idempotent so a crash mid-batch costs seconds of retry instead of hours of re-work; and keeping a live corpus fresh without re-provisioning the entire GPU fleet every time a burst of edits arrives.

Requirements and Constraints

Functional Requirements

• Pull documents from heterogeneous sources (wiki exports, PDF drops in object storage, CMS webhooks, support ticket exports) on a per-source schedule or push trigger
• Split each document into token-bounded, overlap-safe chunks that respect structural boundaries where the source format has them (headers, paragraphs, table rows)
• Deduplicate at both the document level (unchanged re-crawled content) and the chunk level (near-identical chunks from templated content)
• Embed every unique chunk with a versioned embedding model and write the vector plus metadata to the index exactly once
• Detect source document changes and deletions, and drive re-embedding or tombstoning without a full corpus rebuild
• Support resuming a multi-day backfill from the last completed checkpoint after any crash, deploy, or manual pause
• Migrate the entire corpus to a new embedding model version without mixing vector spaces inside one index

Non-Functional Requirements

• Latency (freshness): an edited or new document must be searchable within 15 minutes, p99, in steady state
• Throughput: sustain 1,000 embeddings/second during backfill; ~120 embeddings/second steady state; burst headroom to 2,500/second during a large re-embedding migration
• Quality: zero NaN or malformed vectors reaching the index; chunk boundary accuracy such that fewer than 0.5% of chunks end mid-sentence on structured sources
• Cost: under $1.20 per million chunks embedded, fully loaded (GPU compute, orchestration, storage)
• Capacity: 420 million chunks at 1024 dimensions in float16 is roughly 860GB of raw vectors; metadata and raw chunk text add several hundred more gigabytes in object storage

Constraints

• Embedding model: a 335M-parameter bi-encoder (bge-large-class), 1024 dimensions, served on GPU rather than called through an API, because backfill volume makes per-call API pricing the dominant cost
• Hardware: a GPU pool that autoscales between 2 and 8 replicas, CPU-only fleet for chunking, dedup lookups, and orchestration
• Out of scope: the query-time retrieval path (BM25, HNSW search, reranking) that consumes this index, and multi-modal embedding of images or audio attached to documents. Those are separate system designs that read from the index this pipeline builds.

High-Level Architecture

The system has six major components. The Ingestion Coordinator pulls from every source on its own schedule, tracks a durable checkpoint per source, and emits raw documents exactly once even after a crash. The Chunker splits each document into token-bounded, overlap-safe passages. The Dedup Layer checks a content hash before either stage does real work, skipping documents and chunks that haven’t changed. The Embedding Worker Pool batches unique chunks by token budget and runs GPU inference. The Vector Writer validates every vector and upserts it into the index idempotently, keyed by chunk hash. The Freshness Scheduler watches for source changes and deletions and re-queues affected documents without touching the rest of the corpus.

A document enters through the Ingestion Coordinator, which writes a checkpoint (source ID, cursor position, timestamp) to Postgres before handing the document downstream, so a crash between checkpoint-write and full processing simply means the document gets reprocessed once, not lost. The Chunker and Dedup Layer run as a pair: the dedup check happens on the raw document’s content hash first, and only documents that changed get chunked at all, since chunking a 1,800-token document you’ve already indexed is wasted CPU. Chunks that pass dedup flow into the Embedding Worker Pool, which doesn’t process one chunk at a time but accumulates a batch by token count before running a single GPU forward pass, then hands validated vectors to the Vector Writer for an idempotent upsert.

The timeline below traces one edited document from the moment a change is detected to the moment it’s searchable again, showing where the 15-minute freshness budget actually gets spent.

The Ingestion Coordinator

The coordinator’s job is pulling documents from every source on schedule and guaranteeing each one reaches the pipeline exactly once, even across crashes and deploys.

A smart engineer’s first instinct is to track progress in memory and rely on the source API’s own pagination cursor. This breaks the moment the coordinator process restarts mid-crawl, because in-memory state is gone and the source’s cursor may no longer be valid if the source doesn’t support resumable pagination past a certain window. The coordinator instead writes a durable checkpoint row after every batch it successfully hands off downstream, so a restart resumes exactly where it left off instead of re-crawling from the beginning or, worse, silently skipping a range.

# ingestion_coordinator.py - per-source checkpointed crawl with exactly-once handoff
import hashlib
from dataclasses import dataclass
from datetime import datetime, timezone

@dataclass
class Checkpoint:
    source_id: str
    cursor: str
    last_completed_at: datetime

class IngestionCoordinator:
    def __init__(self, db, source_clients, queue):
        self.db = db
        self.source_clients = source_clients
        self.queue = queue

    async def crawl_source(self, source_id: str, batch_size: int = 500) -> int:
        checkpoint = await self.db.get_checkpoint(source_id)
        client = self.source_clients[source_id]
        documents_handed_off = 0

        async for batch in client.iter_documents(cursor=checkpoint.cursor, batch_size=batch_size):
            for doc in batch.documents:
                content_hash = hashlib.sha256(doc.raw_bytes).hexdigest()
                await self.queue.publish(
                    topic="raw-documents",
                    key=doc.doc_id,
                    value={
                        "doc_id": doc.doc_id,
                        "source_id": source_id,
                        "content_hash": content_hash,
                        "raw_bytes": doc.raw_bytes,
                        "fetched_at": datetime.now(timezone.utc).isoformat(),
                    },
                )
                documents_handed_off += 1

            await self.db.update_checkpoint(
                source_id, cursor=batch.next_cursor, last_completed_at=datetime.now(timezone.utc)
            )

        return documents_handed_off