Every time a user sends their fifth message in a conversation, your LLM server recomputes the attention keys and values for the previous four turns from scratch. Think of it like a chef who, every time a new dish order arrives at a table, goes back to the pantry and re-chops, re-blanches, and re-seasons all the vegetables from earlier courses - even though those components are sitting right there on the pass, ready to use. At a few requests per second, this waste is invisible. At 50,000 concurrent multi-turn sessions, it’s the dominant cost and latency driver in your entire inference stack.

The KV cache is the intermediate activations generated during the transformer’s attention computation - the projected key and value tensors for each token at each layer. On a Llama-3 70B model running on 4xA100 GPUs, a 4,096-token context consumes roughly 4 GB of GPU HBM just for the KV cache. If you have a 20-turn conversation where each turn adds 200 tokens, by turn 20 you’re running a full 4,000-token prefill on every new message - recomputing activations for 19 turns of history that haven’t changed. This is exactly where naive serving engines bleed throughput and latency simultaneously.

The core tension is between memory and compute. GPU HBM is the scarce resource: you need it for model weights, for the KV cache of in-flight tokens, and for the batch of new requests. Storing more conversation history in the KV cache saves prefill FLOPS but steals memory from other requests. Evicting cached history saves memory but forces expensive recomputation. At scale, with 50,000 concurrent sessions each at different turn depths, you need a system that manages KV cache blocks as a shared, addressable pool - not as per-request scratch space that gets discarded after every response.

We need to solve for three things simultaneously: prefix cache lookup that identifies already-computed token sequences, distributed block management that tracks physical GPU memory across replicas, and sticky routing that delivers requests to replicas already holding the relevant cache blocks.

Requirements and Constraints

Functional Requirements

• Cache KV activations for any prefix of a token sequence and serve them on the next matching request
• Support multi-turn conversation sessions where each turn extends the previous turn’s context
• Share cached KV blocks across model replicas when memory pressure is low on a neighbor
• Evict least-recently-used blocks under memory pressure without corrupting in-flight requests
• Invalidate cached blocks when the model version changes or context window policy changes
• Support system prompt caching: system prompts that are identical across thousands of sessions should be computed once and shared globally

Non-Functional Requirements

• TTFT: p50 under 80ms, p99 under 200ms for cache-hit requests (vs 800ms cold for a 4K context)
• Throughput: 50,000 concurrent sessions, 2,000 new turns/second across the cluster
• Cache hit rate: greater than 70% prefix hit rate across all turns in steady state
• Memory efficiency: KV cache layer uses no more than 30% of available HBM per GPU; weights + active batches get the rest
• Cost per million tokens: target below $1.20 fully-loaded (vs $4.80 fully cold)
• Block transfer latency: cross-replica KV block migration completes in under 5ms over NVLink or InfiniBand
• Quality: caching must be bit-exact - reusing a cached block must produce numerically identical output to recomputing it

Constraints

• Model: Llama-3 70B in BF16, tensor-parallel across 4xA100 80GB per replica
• KV block size: fixed at 16 tokens per block (matching PagedAttention’s default)
• Context window: 8,192 tokens maximum; sessions exceeding this trigger sliding window eviction
• Out of scope: cross-session semantic caching (fuzzy prefix match); that’s a different system

High-Level Architecture

The system has six major components layered between the API gateway and the GPU inference workers.

Conversation Router is the entry point. It receives turn N for session S, computes the prefix hash for the conversation history up to turn N-1, and queries the Cache Index to find which replica (if any) holds the matching KV blocks. It then routes the request to the warm replica using sticky consistent hashing - or to any available replica if no warm replica exists.

Cache Index is a distributed in-memory data structure (backed by Redis Cluster) that maps prefix token hashes to the replica ID and physical block addresses holding those blocks. It’s the system’s directory: it knows where every cached prefix lives without holding the cache data itself.

KV Block Manager runs as a sidecar process on each inference replica. It owns the physical HBM allocation for KV blocks, maintains a local radix tree for fast prefix lookup, tracks reference counts so in-flight requests hold blocks open, and evicts LRU blocks when headroom drops below threshold.

Prefill Worker is the GPU process that runs the transformer forward pass to compute KV activations for uncached tokens. It writes new blocks directly to the KV Block Manager. When a request has a partial cache hit (first 3,000 tokens cached, last 500 not), the Prefill Worker computes only the uncached suffix.

Block Transfer Agent handles cross-replica block migration. When the router has no warm local replica for a high-value prefix (system prompts, very-long conversations), the Block Transfer Agent fetches blocks from the source replica over NVLink or InfiniBand and loads them into the destination GPU’s HBM before the prefill starts.

Eviction Oracle is a background service that runs a cost-benefit model: it weighs the probability that a cached block will be used again (based on session activity, turn depth, and historical hit rate) against the HBM cost of holding it. It signals the KV Block Manager when to preemptively evict, before memory pressure forces panic eviction of the wrong blocks.

The KV Block Manager

The KV Block Manager’s job is to turn the GPU’s HBM into a managed pool of fixed-size blocks that can be shared, referenced, and evicted independently.