Build an LLM Inference Gateway with Load Balancing Across Model Replicas Premium

Design an inference gateway that routes 5,000 requests per second across 64 GPU replicas, holds time-to-first-token under 400ms at p99, and picks replicas by KV cache pressure instead of round robin.