A print shop running a single press for both jobs learns this the hard way: the offset run for a 5,000-copy flyer needs the whole machine flat out for twenty minutes, while the letterpress job next to it needs the same machine free for six hours, one impression every few seconds, never fully loaded but never allowed to stop either. Put both jobs on one press and you get the worst of both schedules. LLM inference has the same two jobs living on one GPU, and it has for years: prefill, which reads the entire prompt and computes attention over all of it in one dense, compute-bound burst, and decode, which then generates one token at a time in a loop that is bound almost entirely by how fast you can read the KV cache back out of memory.

Prefill wants big batches and every tensor core saturated. Decode wants low latency per step and enough free memory to hold hundreds of concurrent KV caches. When both run on the same GPU, a single long prompt arriving mid-stream stalls every other user’s decode step behind it, because the scheduler has to finish the compute-heavy prefill pass before it can resume the memory-bound decode pass. This is the co-located serving tax: at 5 requests/second on one A100, nobody notices. At 5,000 requests/second across a shared pool, prefill interference becomes the dominant source of tail latency, and no amount of batching tuning fixes it, because the problem isn’t the scheduler, it’s that one physical GPU is being asked to be two different machines at once.

The fix is architectural, not algorithmic: put prefill and decode on separate GPU pools, sized and provisioned for their own workload shape, and move the computed KV cache between them over a fast interconnect once prefill finishes. Prefill nodes can now run at high batch sizes tuned purely for compute throughput. Decode nodes can now run continuous batching tuned purely for TPOT (time-per-output-token) stability, free of prefill interference. The catch is that this turns a local memory copy into a network transfer, and that transfer has to complete inside the latency budget you were trying to protect in the first place.

We need to solve for three things simultaneously: routing each incoming request to the right prefill node without creating a hot spot, transferring gigabytes of KV cache blocks across the network fast enough that the decode node never idles waiting for them, and sizing the two pools independently so neither one is starved while the other sits idle.

Requirements and Constraints

Functional Requirements

• Accept a completion request, run prefill on a dedicated prefill-tier GPU, and produce the first token and the full KV cache for the prompt
• Transfer the computed KV cache blocks to an assigned decode-tier GPU without re-computing them
• Resume generation on the decode tier using the transferred cache, streaming tokens back to the client via SSE
• Support prompts long enough to require chunked prefill so a single 32k-token prompt does not monopolize a prefill GPU for multiple seconds
• Support multi-turn conversations where a follow-up turn can reuse KV cache already resident on a decode node (prefix cache hit) without a fresh transfer
• Rebalance load across both pools independently as request mix shifts between prefill-heavy (long prompts, short answers) and decode-heavy (short prompts, long answers) traffic

Non-Functional Requirements

• Latency: time-to-first-token (TTFT) p99 under 150ms for prompts up to 2k tokens, including transfer time; time-per-output-token (TPOT) p99 under 30ms once decode starts
• Throughput: sustain 12,000 aggregate decode tokens/second across the decode pool at 500 concurrent sequences average
• Transfer: KV cache transfer must complete in under 40ms for a 2k-token prompt on Llama-3 70B (roughly 900 MB of KV blocks) to stay inside the TTFT budget
• Quality: zero degradation versus co-located serving. Disaggregation is a performance and cost change, not an accuracy change, so output tokens must be bit-identical given the same sampling seed
• Cost: GPU-hours per million output tokens must drop versus co-located serving, since prefill GPUs no longer sit idle waiting for decode loops to catch up
• Capacity: decode-tier GPU memory is the binding constraint, since it holds hundreds of concurrent KV caches; prefill-tier GPU memory only needs to hold the batch currently being processed plus the model weights

Constraints

• Assume Llama-3 70B in FP16 or BF16 as the served model, tensor parallelism degree 4 per replica on both tiers
• Assume 8xH100 80GB nodes with NVLink inside a node and RDMA over InfiniBand (RoCEv2) between nodes
• Assume a transfer engine capable of GPU-direct RDMA (no CPU staging copy) is available, either a custom implementation or an existing one such as NIXL or Mooncake’s transfer engine
• Out of scope: multi-modal inputs, speculative decoding, and fine-tuning workloads. This design is serving-only
• Out of scope: cross-region disaggregation. Prefill and decode pools live in the same data center on the same RDMA fabric

High-Level Architecture

The system has five major components. The Global Scheduler accepts requests and assigns each one to a prefill node and, once prefill finishes, to a decode node. The Prefill Pool is a set of GPU replicas tuned for high-batch compute throughput, running chunked prefill so long prompts don’t block the queue. The Transfer Engine moves KV cache blocks from a prefill node’s HBM directly into a decode node’s HBM over RDMA, without a CPU-side copy. The Decode Pool is a separate set of GPU replicas running continuous batching, receiving already-computed KV caches and generating tokens against them. The KV Block Directory is a metadata service, backed by Redis, that tracks which physical GPU and block offsets currently own each sequence’s KV cache, so the scheduler and transfer engine always know where data lives.

A request arrives at the Global Scheduler, which checks the KV Block Directory for a prefix cache hit (a returning conversation whose earlier turns are still resident on some decode node), then assigns the request to the least-loaded prefill replica. The prefill replica runs the forward pass over the prompt, writes KV blocks into its own paged memory pool, and signals completion. The Transfer Engine then pulls those blocks directly across RDMA into a decode replica chosen by the scheduler for having headroom in its KV cache budget, updates the KV Block Directory with the new owner, and hands control to the decode replica’s continuous batching loop, which starts streaming tokens back to the client over SSE.

The Global Scheduler

The scheduler’s job is to decide, for every request, which prefill node computes it and which decode node continues it, without letting either pool develop a hot spot.

A naive round-robin assignment looks fine until traffic mixes long prompts with short ones. A prefill node that just got handed three 16k-token prompts in a row is now the slowest node in the pool for the next several seconds, and round-robin keeps sending it more work anyway because it doesn’t know that.