AI System Design - scalability
#01 Performance
Build an LLM Cold Start Optimizer for Serverless Inference Premium
How to cut LLM serverless cold starts from 60-90 seconds to under 4 seconds using snapshotted model weights, warm pools, and predictive pre-provisioning, so a scale-to-zero GPU fleet absorbs bursty traffic without breaching a 2-second p99 time-to-first-token SLA.
Read
#02 Performance
Build an LLM Inference Gateway with Load Balancing Across Model Replicas Premium
Design an inference gateway that routes 5,000 requests per second across 64 GPU replicas, holds time-to-first-token under 400ms at p99, and picks replicas by KV cache pressure instead of round robin.
Read