Think of a card catalog in a library from the 1970s. You want to find books “similar to” a book you already read. A keyword search looks for exact word matches - title, author, subject heading. But similarity is about meaning, not words. Vector search is the modern equivalent of a librarian who has read everything and can say “based on the themes in that novel, you’d also like these three, even though their titles share no words.”

The engineering challenge is that measuring similarity between 100 million vectors of 1,536 dimensions - typical for a modern embedding model - requires computing dot products or cosine distances. Exact nearest neighbor search requires comparing a query against all 100 million vectors: even at 1 nanosecond per distance calculation, that is 100 milliseconds of pure compute per query. At 10,000 queries per second that is 1,000 CPU-seconds per second - a compute bill that scales linearly with dataset size.

Approximate Nearest Neighbor (ANN) algorithms solve this by accepting a small accuracy penalty in exchange for orders-of-magnitude speedup. The HNSW (Hierarchical Navigable Small World) graph structure can search 100 million 1,536-dimensional vectors in under 2ms with 95%+ recall. But HNSW introduces new challenges: the graph is built in memory, making it expensive at scale; updates to the index require rebuilding affected graph connections; and running HNSW at production scale means sharding, replication, and live updates without search downtime.

Quantization further reduces memory - a 1,536-dimension float32 vector is 6KB; the same vector quantized to int8 is 1.5KB, and a binary quantized version is 192 bytes. But quantization degrades recall, so you re-rank the approximate candidates against the full-precision vectors to recover accuracy.

We need to solve for ANN index structure, live index updates, quantization with re-ranking, and horizontal sharding simultaneously.

Requirements and Constraints

Functional Requirements

• Index 100 million high-dimensional embedding vectors (1,024 to 1,536 dimensions, float32)
• Serve k-nearest-neighbor queries (k=1 to 100) with under 20ms p99 latency
• Support live index updates: insert, update, and soft-delete vectors without taking the index offline
• Metadata filtering: combine vector similarity with structured filters (e.g., “find the 10 most similar documents that are also tagged category=finance and date > 2024-01-01”)
• Batch indexing: ingest up to 1 million new vectors per hour during bulk loads

Non-Functional Requirements

• Query latency: p50 under 5ms, p99 under 20ms, p999 under 50ms
• Recall at k=10: above 95% vs exact nearest neighbor
• Throughput: 10,000 queries/second at target latency
• Availability: 99.99% for reads; 99.9% for writes
• Storage: 100 million x 1,536 x 4 bytes = 614 GB raw; target under 200 GB with quantization
• Index rebuild time: under 4 hours for full 100M vector rebuild

Constraints

• Vector dimensionality is fixed at index creation; schema changes require full rebuild
• Approximate results are acceptable - exact KNN is out of scope for production queries
• Cross-shard filtering requires merging results; we handle this at the query router layer

High-Level Architecture

The system has four planes: the ingestion pipeline that embeds and indexes new vectors, the index layer that maintains the HNSW graph per shard, the query router that fans out searches and merges results, and the re-ranking layer that refines approximate results.

Raw documents arrive at the embedding service, which calls an embedding model (OpenAI, Cohere, or a self-hosted sentence-transformer) to produce dense vectors. Vectors are routed to the correct index shard based on a hash of the vector ID. Each shard maintains an in-memory HNSW graph and a persistent vector store on disk. At query time, the query router converts the query text to an embedding, fans out the search to all shards, collects top-k candidates from each, and merges them. The re-ranker optionally scores the merged candidates against the original query for precision-critical use cases.