A valet garage that shuts its elevator off between customers saves real electricity. Nobody minds, as long as fetching the car still takes ten seconds. If fetching the car instead takes ninety, drivers stop using the valet and just circle the block themselves. Serverless GPU inference makes exactly that trade: scale a model’s replica count to zero when nobody is asking it anything, and only pay for the elevator when a car is actually requested. The problem is that loading a large language model onto a GPU from a cold start is nothing like calling an elevator. It means pulling a container image, initializing CUDA, fetching tens of gigabytes of weights from object storage, and warming up an allocator, and for a 13-billion-parameter model that whole sequence commonly takes 60 to 90 seconds.

A user who just typed a prompt is not going to wait a minute and a half for the first token, and no product team will ship a chat feature with that latency profile. The naive instinct is to just keep replicas warm all the time, but a 13B model in FP16 needs roughly 26GB of GPU memory resident, and paying for that 24 hours a day on hardware that sits idle outside business hours, weekday mornings, or the ten minutes after a marketing email goes out, throws away the entire economic point of serverless. The naive instinct is to just prefetch weights to local disk instead, but that only removes the object storage fetch penalty; the CUDA initialization, the tensor materialization onto the GPU, and the allocator warmup are still paid in full, and on a genuinely first-of-a-kind cold node there is still 26GB of bytes that has to move before anything can run. The naive instinct is to just keep a fixed minimum number of replicas warm, but real traffic is bursty and directional: a support chatbot spikes at 9am, a coding assistant spikes after a product launch, and a fixed number is either wrong most of the time in the expensive direction or wrong at the worst possible moment in the latency-violating direction.

The forces in tension are latency, cost, and uncertainty, and none of them can be optimized in isolation. Latency wants every replica warm, all the time. Cost wants every idle GPU-second eliminated. Uncertainty means the traffic that determines which of those wins on any given minute is not known in advance with certainty, only with a forecast that gets less reliable the further out you look. We need to solve for three things simultaneously: how to make a genuinely cold replica ready to serve in low single-digit seconds instead of a minute and a half, how many replicas to keep warm ahead of demand without paying for capacity nobody uses, and how to absorb the gap between “traffic arrived” and “a replica is ready” without dropping requests or silently blowing the latency budget.

Requirements and Constraints

Functional Requirements

• Detect, per incoming request, whether a warm replica with spare capacity exists; if not, route the request to a bounded-wait cold-start path instead of blocking indefinitely
• Snapshot a freshly warmed replica’s full process and GPU memory state, once, after its first cold boot, and persist that snapshot to a local NVMe cache and to a shared regional object store
• Restore a snapshot onto a freshly allocated GPU in low single-digit seconds, reproducing the exact state (weights, KV cache allocator, CUDA context) a naturally warm replica would already have
• Maintain a per-model minimum warm replica count that adapts to a rolling traffic forecast rather than a fixed static number
• Queue requests that arrive with no warm capacity available, bounded by a maximum wait time, and shed load past that bound instead of letting the queue grow without limit
• Scale a replica down to zero after an idle timeout, but only after confirming a fresh snapshot exists on durable storage, so the next cold start for that model is fast

Non-Functional Requirements

• Latency: time-to-first-token (TTFT) p99 under 2,000ms for a warm-hit request; TTFT p99 under 4,000ms for a request served via snapshot restore, against a 60,000 to 90,000ms baseline for an uncached cold boot
• Throughput: absorb a burst of up to 10x baseline request volume (20 req/s to 200 req/s) within one snapshot-restore window without the admission queue exceeding its maximum wait bound
• Quality: a restored replica must produce token-for-token identical output to a natively warm replica for the same input and sampling seed; a snapshot is a checkpoint of existing state, not a re-derivation of it
• Cost: aggregate GPU-hours billed for idle warm capacity should stay under roughly 20% of what a fleet permanently sized for peak traffic would cost
• Capacity: local NVMe snapshot cache per node must hold at least 2 to 3 model snapshot variants at roughly 28GB each, with enough headroom left over for the KV cache the restored replica needs once serving

Constraints

• Assume a 13B-parameter dense transformer (Llama-2-13B-class), FP16 weights at roughly 26GB, running unsharded on a single GPU per replica, 40 transformer layers, 8 KV heads via grouped-query attention, head dimension 128
• Assume A100 40GB GPUs per node, with local NVMe storage supporting GPUDirect Storage for direct NVMe-to-GPU transfer, and an S3-class object store holding the canonical snapshot and checkpoint artifacts
• Assume the serving stack already implements continuous batching and a KV cache allocator; this design treats that stack as the payload being snapshotted, not something it re-implements
• Out of scope: multi-GPU sharded models. A model that needs eight GPUs to hold its weights has a fundamentally different restore problem, since the snapshot spans multiple devices and an interconnect
• Out of scope: cross-region failover and multi-cloud snapshot portability. Assume one region, one object store, one GPU generation, and one driver version per node pool

High-Level Architecture

The system has six major components. The Cold Start Predictor watches per-model traffic signals and forecasts demand far enough ahead to act before a burst arrives, not after. The Warm Pool Scheduler owns the actual scale-out and scale-in decisions, translating the predictor’s forecast into a target replica count and reconciling it against reality. The Snapshot Manager captures a warmed replica’s full GPU and process state into a portable artifact, and restores that artifact onto a fresh GPU in seconds. The Weight Cache and NVMe Loader keeps model weights staged on fast local storage and supports lazy, layer-by-layer materialization for the rare case where nothing is cached yet. The Admission Controller buffers requests that arrive with no warm capacity available, bounded by a wait-time budget, and sheds load past that bound. The Replica Router is the front door, deciding in milliseconds whether a request can go straight to a warm replica or has to enter the cold path.

A request lands at the Replica Router, which checks whether any warm replica for the requested model has spare KV cache headroom. If one does, the request goes straight to it and streams tokens back over SSE within the warm-hit latency budget. If none does, the router hands the request to the Admission Controller, which enqueues it with a deadline, and signals the Snapshot Manager to restore a replica. The Snapshot Manager pulls the most recent snapshot for that model from the local NVMe Snapshot Store, verifies its checksum and driver version, and resumes the process directly into a state that is indistinguishable from a replica that never went cold.

Running in the background, the Cold Start Predictor continuously ingests recent QPS and forecasts demand a lead time ahead, and the Warm Pool Scheduler uses that forecast to pre-warm replicas before the admission queue ever has to absorb a request. The Weight Cache and NVMe Loader is the fallback for the one case snapshotting cannot cover on its own: the very first cold boot of a brand-new model version, which has no prior snapshot to restore from.

The Cold Start Predictor

The predictor’s job is to forecast near-future request volume per model and translate that forecast into a target warm replica count, far enough ahead that a replica is ready before demand actually arrives.

The non-obvious part: reacting to a threshold crossing is already too slow. If the trigger is “queue depth exceeded five requests,” that signal only fires after requests are already waiting, and by the time a fresh replica is restored the queue has been growing for the entire 3.6 second restore window on top of whatever it takes to notice and act. The predictor has to work off leading indicators, current trend and rate of change, not lagging ones like queue depth or GPU utilization.

# Cold start predictor: Holt's double exponential smoothing forecasts qps `lead_time_s` ahead
import math
from dataclasses import dataclass
from collections import deque

@dataclass
class ForecastState:
    level: float  # smoothed qps level
    trend: float  # smoothed qps trend per tick

class ColdStartPredictor:
    def __init__(self, alpha: float = 0.35, beta: float = 0.15, lead_time_s: float = 15.0):
        self.alpha = alpha
        self.beta = beta
        self.lead_time_s = lead_time_s
        self.state: ForecastState | None = None
        self.history: deque[float] = deque(maxlen=120)

    def observe(self, qps_sample: float) -> None:
        # call once per second with the latest observed qps for this model
        self.history.append(qps_sample)
        if self.state is None:
            self.state = ForecastState(level=qps_sample, trend=0.0)
            return
        prev_level = self.state.level
        level = self.alpha * qps_sample + (1 - self.alpha) * (prev_level + self.state.trend)
        trend = self.beta * (level - prev_level) + (1 - self.beta) * self.state.trend
        self.state = ForecastState(level=level, trend=trend)

    def forecast_qps(self) -> float:
        if self.state is None:
            return 0.0
        return max(0.0, self.state.level + self.lead_time_s * self.state.trend)

    def target_replica_count(self, seqs_per_replica: int, avg_service_time_s: float, target_util: float = 0.75) -> int:
        forecast = self.forecast_qps()
        in_flight = forecast * avg_service_time_s
        needed = in_flight / (seqs_per_replica * target_util)
        return max(1, math.ceil(needed))