A valet garage that shuts its elevator off between customers saves real electricity. Nobody minds, as long as fetching the car still takes ten seconds. If fetching the car instead takes ninety, drivers stop using the valet and just circle the block themselves. Serverless GPU inference makes exactly that trade: scale a model’s replica count to zero when nobody is asking it anything, and only pay for the elevator when a car is actually requested. The problem is that loading a large language model onto a GPU from a cold start is nothing like calling an elevator. It means pulling a container image, initializing CUDA, fetching tens of gigabytes of weights from object storage, and warming up an allocator, and for a 13-billion-parameter model that whole sequence commonly takes 60 to 90 seconds.

A user who just typed a prompt is not going to wait a minute and a half for the first token, and no product team will ship a chat feature with that latency profile. The naive instinct is to just keep replicas warm all the time, but a 13B model in FP16 needs roughly 26GB of GPU memory resident, and paying for that 24 hours a day on hardware that sits idle outside business hours, weekday mornings, or the ten minutes after a marketing email goes out, throws away the entire economic point of serverless. The naive instinct is to just prefetch weights to local disk instead, but that only removes the object storage fetch penalty; the CUDA initialization, the tensor materialization onto the GPU, and the allocator warmup are still paid in full, and on a genuinely first-of-a-kind cold node there is still 26GB of bytes that has to move before anything can run. The naive instinct is to just keep a fixed minimum number of replicas warm, but real traffic is bursty and directional: a support chatbot spikes at 9am, a coding assistant spikes after a product launch, and a fixed number is either wrong most of the time in the expensive direction or wrong at the worst possible moment in the latency-violating direction.

The forces in tension are latency, cost, and uncertainty, and none of them can be optimized in isolation. Latency wants every replica warm, all the time. Cost wants every idle GPU-second eliminated. Uncertainty means the traffic that determines which of those wins on any given minute is not known in advance with certainty, only with a forecast that gets less reliable the further out you look. We need to solve for three things simultaneously: how to make a genuinely cold replica ready to serve in low single-digit seconds instead of a minute and a half, how many replicas to keep warm ahead of demand without paying for capacity nobody uses, and how to absorb the gap between “traffic arrived” and “a replica is ready” without dropping requests or silently blowing the latency budget.

Requirements and Constraints

Functional Requirements

• Detect, per incoming request, whether a warm replica with spare capacity exists; if not, route the request to a bounded-wait cold-start path instead of blocking indefinitely
• Snapshot a freshly warmed replica’s full process and GPU memory state, once, after its first cold boot, and persist that snapshot to a local NVMe cache and to a shared regional object store
• Restore a snapshot onto a freshly allocated GPU in low single-digit seconds, reproducing the exact state (weights, KV cache allocator, CUDA context) a naturally warm replica would already have
• Maintain a per-model minimum warm replica count that adapts to a rolling traffic forecast rather than a fixed static number
• Queue requests that arrive with no warm capacity available, bounded by a maximum wait time, and shed load past that bound instead of letting the queue grow without limit
• Scale a replica down to zero after an idle timeout, but only after confirming a fresh snapshot exists on durable storage, so the next cold start for that model is fast

Non-Functional Requirements

• Latency: time-to-first-token (TTFT) p99 under 2,000ms for a warm-hit request; TTFT p99 under 4,000ms for a request served via snapshot restore, against a 60,000 to 90,000ms baseline for an uncached cold boot
• Throughput: absorb a burst of up to 10x baseline request volume (20 req/s to 200 req/s) within one snapshot-restore window without the admission queue exceeding its maximum wait bound
• Quality: a restored replica must produce token-for-token identical output to a natively warm replica for the same input and sampling seed; a snapshot is a checkpoint of existing state, not a re-derivation of it
• Cost: aggregate GPU-hours billed for idle warm capacity should stay under roughly 20% of what a fleet permanently sized for peak traffic would cost
• Capacity: local NVMe snapshot cache per node must hold at least 2 to 3 model snapshot variants at roughly 28GB each, with enough headroom left over for the KV cache the restored replica needs once serving

Constraints

• Assume a 13B-parameter dense transformer (Llama-2-13B-class), FP16 weights at roughly 26GB, running unsharded on a single GPU per replica, 40 transformer layers, 8 KV heads via grouped-query attention, head dimension 128
• Assume A100 40GB GPUs per node, with local NVMe storage supporting GPUDirect Storage for direct NVMe-to-GPU transfer, and an S3-class object store holding the canonical snapshot and checkpoint artifacts
• Assume the serving stack already implements continuous batching and a KV cache allocator; this design treats that stack as the payload being snapshotted, not something it re-implements
• Out of scope: multi-GPU sharded models. A model that needs eight GPUs to hold its weights has a fundamentally different restore problem, since the snapshot spans multiple devices and an interconnect
• Out of scope: cross-region failover and multi-cloud snapshot portability. Assume one region, one object store, one GPU generation, and one driver version per node pool

High-Level Architecture

The system has six major components. The Cold Start Predictor watches per-model traffic signals and forecasts demand far enough ahead to act before a burst arrives, not after. The Warm Pool Scheduler owns the actual scale-out and scale-in decisions, translating the predictor’s forecast into a target replica count and reconciling it against reality. The Snapshot Manager captures a warmed replica’s full GPU and process state into a portable artifact, and restores that artifact onto a fresh GPU in seconds. The Weight Cache and NVMe Loader keeps model weights staged on fast local storage and supports lazy, layer-by-layer materialization for the rare case where nothing is cached yet. The Admission Controller buffers requests that arrive with no warm capacity available, bounded by a wait-time budget, and sheds load past that bound. The Replica Router is the front door, deciding in milliseconds whether a request can go straight to a warm replica or has to enter the cold path.

A request lands at the Replica Router, which checks whether any warm replica for the requested model has spare KV cache headroom. If one does, the request goes straight to it and streams tokens back over SSE within the warm-hit latency budget. If none does, the router hands the request to the Admission Controller, which enqueues it with a deadline, and signals the Snapshot Manager to restore a replica. The Snapshot Manager pulls the most recent snapshot for that model from the local NVMe Snapshot Store, verifies its checksum and driver version, and resumes the process directly into a state that is indistinguishable from a replica that never went cold.

Running in the background, the Cold Start Predictor continuously ingests recent QPS and forecasts demand a lead time ahead, and the Warm Pool Scheduler uses that forecast to pre-warm replicas before the admission queue ever has to absorb a request. The Weight Cache and NVMe Loader is the fallback for the one case snapshotting cannot cover on its own: the very first cold boot of a brand-new model version, which has no prior snapshot to restore from.

The Cold Start Predictor

The predictor’s job is to forecast near-future request volume per model and translate that forecast into a target warm replica count, far enough ahead that a replica is ready before demand actually arrives.

The non-obvious part: reacting to a threshold crossing is already too slow. If the trigger is “queue depth exceeded five requests,” that signal only fires after requests are already waiting, and by the time a fresh replica is restored the queue has been growing for the entire 3.6 second restore window on top of whatever it takes to notice and act. The predictor has to work off leading indicators, current trend and rate of change, not lagging ones like queue depth or GPU utilization.

# Cold start predictor: Holt's double exponential smoothing forecasts qps `lead_time_s` ahead
import math
from dataclasses import dataclass
from collections import deque

@dataclass
class ForecastState:
    level: float  # smoothed qps level
    trend: float  # smoothed qps trend per tick

class ColdStartPredictor:
    def __init__(self, alpha: float = 0.35, beta: float = 0.15, lead_time_s: float = 15.0):
        self.alpha = alpha
        self.beta = beta
        self.lead_time_s = lead_time_s
        self.state: ForecastState | None = None
        self.history: deque[float] = deque(maxlen=120)

    def observe(self, qps_sample: float) -> None:
        # call once per second with the latest observed qps for this model
        self.history.append(qps_sample)
        if self.state is None:
            self.state = ForecastState(level=qps_sample, trend=0.0)
            return
        prev_level = self.state.level
        level = self.alpha * qps_sample + (1 - self.alpha) * (prev_level + self.state.trend)
        trend = self.beta * (level - prev_level) + (1 - self.beta) * self.state.trend
        self.state = ForecastState(level=level, trend=trend)

    def forecast_qps(self) -> float:
        if self.state is None:
            return 0.0
        return max(0.0, self.state.level + self.lead_time_s * self.state.trend)

    def target_replica_count(self, seqs_per_replica: int, avg_service_time_s: float, target_util: float = 0.75) -> int:
        forecast = self.forecast_qps()
        in_flight = forecast * avg_service_time_s
        needed = in_flight / (seqs_per_replica * target_util)
        return max(1, math.ceil(needed))

The analogy is a call center staffing the next shift off the forecasted call volume for the next fifteen minutes, not the call volume from fifteen minutes ago. What breaks if this is simplified to a static threshold rule: either the threshold is set conservatively and the fleet stays over-provisioned most of the day, or it is set aggressively and every burst outruns it by exactly the amount of time a threshold-based rule needs to notice a trend has started.

The Warm Pool Scheduler

The scheduler’s job is to reconcile the predictor’s target replica count against the replicas that actually exist right now, scaling out when the target rises and scaling in, carefully, when it falls.

The non-obvious part: scaling in is not the mirror image of scaling out. A replica cannot be safely killed the moment it looks idle, because if its post-boot snapshot has not finished writing to durable storage yet, killing it throws away the one thing that makes the next cold start for that model fast. Scale-in has to wait on snapshot durability, not just an idle timer.

// Warm pool scheduler: reconciles target replica count against actual warm replicas
package scheduler

import (
	"context"
	"time"
)

type ReplicaState string

const (
	StateWarm      ReplicaState = "warm"
	StateRestoring ReplicaState = "restoring"
	StateDraining  ReplicaState = "draining"
)

type Replica struct {
	ID            string
	ModelName     string
	State         ReplicaState
	SnapshotReady bool
	LastActiveAt  time.Time
}

type Scheduler struct {
	Replicas    map[string]*Replica
	IdleTimeout time.Duration
}

// Reconcile scales out to meet target immediately, but never scales in a replica
// before its post-boot snapshot has finished writing to durable storage.
func (s *Scheduler) Reconcile(ctx context.Context, modelName string, target int, scaleOut func(context.Context, string) error, snapshotAndDrain func(context.Context, *Replica) error) error {
	warm := s.warmReplicas(modelName)
	if len(warm) < target {
		for i := 0; i < target-len(warm); i++ {
			if err := scaleOut(ctx, modelName); err != nil {
				return err
			}
		}
		return nil
	}
	for _, r := range warm {
		if len(warm) <= target {
			break
		}
		if time.Since(r.LastActiveAt) <= s.IdleTimeout {
			continue
		}
		if !r.SnapshotReady {
			continue // never scale down a replica with no durable snapshot yet
		}
		if err := snapshotAndDrain(ctx, r); err != nil {
			return err
		}
	}
	return nil
}

func (s *Scheduler) warmReplicas(modelName string) []*Replica {
	var out []*Replica
	for _, r := range s.Replicas {
		if r.ModelName == modelName && r.State == StateWarm {
			out = append(out, r)
		}
	}
	return out
}

The Snapshot Manager

The snapshot manager’s job is to capture a fully warmed process, CUDA context and GPU memory contents included, into a portable artifact, and to restore that artifact onto a new GPU in seconds instead of minutes.

The non-obvious part: standard Linux checkpoint and restore tooling like CRIU understands regular process memory, but it does not natively understand memory that lives inside a GPU’s own address space, managed by the CUDA driver rather than the kernel. Getting a GPU process into a checkpointable state requires an explicit hook that flushes and re-imports device memory through the driver, which is what dedicated checkpoint utilities exist to do.

# Snapshot manager: captures a warmed replica's GPU + process state to NVMe
import time
import json
import hashlib
from pathlib import Path
import cuda_checkpoint  # vendor bindings around the CUDA driver's checkpoint/restore API
import criu

SNAPSHOT_ROOT = Path("/mnt/nvme/snapshots")

def take_snapshot(pid: int, model_name: str, driver_version: str) -> dict:
    snapshot_dir = SNAPSHOT_ROOT / model_name / str(int(time.time()))
    snapshot_dir.mkdir(parents=True, exist_ok=True)

    cuda_checkpoint.toggle(pid=pid)  # freezes GPU execution, flushes device memory to host
    criu.dump(tree=pid, images_dir=str(snapshot_dir), leave_stopped=True)

    digest = _hash_dir(snapshot_dir)
    manifest = {
        "model_name": model_name,
        "driver_version": driver_version,
        "created_at": time.time(),
        "path": str(snapshot_dir),
        "sha256": digest,
        "size_bytes": _dir_size(snapshot_dir),
    }
    (snapshot_dir / "manifest.json").write_text(json.dumps(manifest))
    return manifest

def restore_snapshot(manifest: dict, local_driver_version: str) -> None:
    if manifest["driver_version"] != local_driver_version:
        raise RuntimeError(
            f"driver mismatch: snapshot taken on {manifest['driver_version']}, "
            f"node runs {local_driver_version}, refusing restore"
        )
    snapshot_dir = Path(manifest["path"])
    actual = _hash_dir(snapshot_dir)
    if actual != manifest["sha256"]:
        raise RuntimeError(f"snapshot checksum mismatch: expected {manifest['sha256']}, got {actual}")
    criu.restore(images_dir=str(snapshot_dir), restore_detached=True)

def _dir_size(path: Path) -> int:
    return sum(f.stat().st_size for f in path.rglob("*") if f.is_file())

def _hash_dir(path: Path) -> str:
    digest = hashlib.sha256()
    for f in sorted(path.rglob("*")):
        if f.is_file():
            digest.update(f.read_bytes())
    return digest.hexdigest()

What would break if this were simplified to “just re-run the normal boot sequence, but with weights already on local disk”: the CUDA driver initialization, the allocator warmup, and the first forward pass that primes kernel caches would all still run in full, because none of that state is captured by having the raw weight bytes sitting on NVMe. A snapshot captures the process after all of that has already happened once.

The Weight Cache and NVMe Loader

The loader’s job is to make sure that by the time a snapshot restore or a genuine cold boot needs the model’s weights, those weights are already sitting on fast local storage rather than something that has to be pulled from object storage on the request path.

The non-obvious part: even with snapshotting fully in place, the very first replica of a brand-new model version has nothing to restore from, and that path still has to move bytes. Blocking until all 26GB has landed on the GPU wastes the fact that the first few transformer layers could start computing the moment their own weights are resident, without waiting on the rest.

# Lazy weight loader: mmaps the safetensors file, materializes layers on demand so the
# first transformer layers can start computing before the full 26GB has landed on GPU
import torch
from safetensors import safe_open

class LazyLayerLoader:
    def __init__(self, shard_path: str, device: str = "cuda:0"):
        self._handle = safe_open(shard_path, framework="pt", device="cpu")  # mmap, no eager copy
        self.device = device
        self._resident: dict[str, dict[str, torch.Tensor]] = {}

    def get_layer(self, layer_prefix: str) -> dict[str, torch.Tensor]:
        if layer_prefix in self._resident:
            return self._resident[layer_prefix]
        tensors = {}
        for key in self._handle.keys():
            if key.startswith(layer_prefix):
                tensors[key] = self._handle.get_tensor(key).to(self.device, non_blocking=True)
        self._resident[layer_prefix] = tensors
        return tensors

    def prefetch_ahead(self, layer_prefixes: list[str], ahead: int = 2) -> None:
        # kicks off loads for the next few layers while the current one computes
        for prefix in layer_prefixes[:ahead]:
            if prefix not in self._resident:
                self.get_layer(prefix)

The analogy is streaming a video instead of waiting for the whole file to download. Only the frame about to play needs to be ready, not the ending. Once a model has been through this path once, its weights are staged locally and its snapshot exists, so every subsequent cold start for that model uses the fast restore path instead.

The Admission Controller and Request Queue

The admission controller’s job is to hold requests that arrive with no warm capacity available for at most as long as a restore is expected to take, and to shed load past that bound rather than let a queue grow without limit.

// Admission controller: bounded-wait queue for requests arriving during a cold-start restore
package admission

import (
	"container/heap"
	"errors"
	"sync"
	"time"
)

type QueuedRequest struct {
	ID       string
	Deadline time.Time
	index    int
}

type deadlineHeap []*QueuedRequest

func (h deadlineHeap) Len() int           { return len(h) }
func (h deadlineHeap) Less(i, j int) bool { return h[i].Deadline.Before(h[j].Deadline) }
func (h deadlineHeap) Swap(i, j int)      { h[i], h[j] = h[j], h[i]; h[i].index = i; h[j].index = j }
func (h *deadlineHeap) Push(x any)        { *h = append(*h, x.(*QueuedRequest)) }
func (h *deadlineHeap) Pop() any {
	old := *h
	n := len(old)
	item := old[n-1]
	*h = old[:n-1]
	return item
}

var ErrQueueFull = errors.New("admission queue full, shedding load")

type Controller struct {
	mu       sync.Mutex
	queue    deadlineHeap
	maxDepth int
	maxWait  time.Duration
}

func (c *Controller) Enqueue(id string) (*QueuedRequest, error) {
	c.mu.Lock()
	defer c.mu.Unlock()
	if len(c.queue) >= c.maxDepth {
		return nil, ErrQueueFull
	}
	req := &QueuedRequest{ID: id, Deadline: time.Now().Add(c.maxWait)}
	heap.Push(&c.queue, req)
	return req, nil
}

func (c *Controller) DrainExpired() []*QueuedRequest {
	c.mu.Lock()
	defer c.mu.Unlock()
	var expired []*QueuedRequest
	now := time.Now()
	for len(c.queue) > 0 && c.queue[0].Deadline.Before(now) {
		expired = append(expired, heap.Pop(&c.queue).(*QueuedRequest))
	}
	return expired
}

Without a bounded queue, a burst simply gets rejected at the door the instant the last warm replica takes its final request. With an unbounded queue, one stuck restore, a driver mismatch or an NVMe path that went unavailable, can pile thousands of requests behind a single slot that will never open.

The Replica Router

The router’s job is to decide, on the critical path of every request, whether a warm replica with spare capacity already exists or the request needs to go through admission control.

// Replica router: sends a request straight to a warm replica, or hands it to admission control
package router

import (
	"errors"
	"sync"
)

type Replica struct {
	ID    string
	State string // "warm", "restoring", "cold"
	Load  int
}

type Router struct {
	mu       sync.Mutex
	replicas []*Replica
}

var ErrNoWarmReplica = errors.New("no warm replica available, route to admission control")

func (r *Router) RouteToWarm(modelName string) (*Replica, error) {
	r.mu.Lock()
	defer r.mu.Unlock()

	var best *Replica
	for _, rep := range r.replicas {
		if rep.State != "warm" {
			continue
		}
		if best == nil || rep.Load < best.Load {
			best = rep
		}
	}
	if best == nil {
		return nil, ErrNoWarmReplica
	}
	best.Load++
	return best, nil
}

Put together, the warm-hit path and the cold-start path look like two different systems from the outside, but they share every component except which fork the router takes at the very first decision point.

Data Model

The system tracks three kinds of state: relational metadata about replicas and their backing snapshots, a hot-path readiness registry, and cold-start telemetry events for observability.

-- Snapshot manifest: one row per captured GPU memory snapshot, pinned to a driver version
CREATE TABLE snapshots (
    snapshot_id       UUID PRIMARY KEY,
    model_name        TEXT NOT NULL,
    driver_version    TEXT NOT NULL,
    size_bytes        BIGINT NOT NULL,
    sha256            TEXT NOT NULL,
    object_store_uri  TEXT NOT NULL,
    status            TEXT NOT NULL CHECK (status IN ('pending', 'ready', 'corrupt', 'stale')),
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_snapshots_model ON snapshots (model_name, status, created_at DESC);

-- Replica registry: tracks every GPU replica's lifecycle state and which snapshot restored it
CREATE TABLE replicas (
    replica_id       UUID PRIMARY KEY,
    model_name       TEXT NOT NULL,
    node_id          TEXT NOT NULL,
    state            TEXT NOT NULL CHECK (state IN ('cold', 'restoring', 'warm', 'draining', 'dead')),
    snapshot_id      UUID REFERENCES snapshots(snapshot_id),
    last_active_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_replicas_model_state ON replicas (model_name, state);

-- Traffic forecast: rolling per-model qps forecast consumed by the warm pool scheduler
CREATE TABLE traffic_forecasts (
    model_name       TEXT NOT NULL,
    window_start     TIMESTAMPTZ NOT NULL,
    forecast_qps     DOUBLE PRECISION NOT NULL,
    actual_qps       DOUBLE PRECISION,
    target_replicas  INT NOT NULL,
    PRIMARY KEY (model_name, window_start)
);

Cold-start telemetry is high-volume and short-lived, emitted on every request that touches the router or the admission controller, so it is modeled as an event rather than a row that gets updated:

// Cold start event: emitted by the router and admission controller on every path decision
syntax = "proto3";

package aisd.coldstart;

message ColdStartEvent {
  string request_id = 1;
  string model_name = 2;
  string path = 3;          // "warm_hit", "snapshot_restore", "cold_boot"
  double queue_wait_ms = 4;
  double restore_ms = 5;
  double ttft_ms = 6;
  bool sla_breached = 7;
  int64 timestamp_ms = 8;
}

The readiness registry sits in Redis rather than the relational store, since the router and admission controller both need sub-millisecond lookups on the request path:

HSET replica:warm-pool:llama2-13b replica-04 warm replica-07 restoring replica-11 warm
EXPIRE replica:warm-pool:llama2-13b 30
ZADD replica_load 0.42 "replica-04"

Partitioning key choice is model_name for traffic_forecasts, since forecasting is inherently per-model, and replica_id for replicas, since a replica’s lifecycle belongs to exactly one node and one snapshot at a time.

Key Algorithms and Protocols

Holt’s Double Exponential Smoothing for Traffic Forecasting

Already covered in the Cold Start Predictor section above. The property that makes this forecast usable, rather than merely present, is that its lead time has to exceed the sum of restore latency and scheduler decision latency, or the forecast is structurally too late no matter how accurate it is. A perfectly accurate 2-second forecast is useless against a 3.6 second restore.

GPU Memory Snapshot and Restore

Already covered in the Snapshot Manager section above. The edge case worth naming explicitly: a partial or corrupted restore must never be allowed to serve traffic. The checksum and driver version check exist specifically to fail closed, falling back to a full cold boot, rather than resuming a process into an undefined state that happens to not crash.

Deadline-Aware Admission Control

Already covered in the Admission Controller section above. Time complexity per enqueue and dequeue is O(log n) via the heap. The edge case is starvation under sustained overload: a naive FIFO queue with a fixed wait bound will admit newer requests that still have deadline headroom while older ones expire, which is correct behavior, not a bug, as long as the deadline bound itself is set from the restore latency’s p99, not its average.

Optimal Keep-Alive Window

The property that makes a keep-alive timeout correct, rather than a guessed constant, is that it should be the breakeven point where one more second of idle GPU cost equals the expected savings from avoiding a cold start in that same second.

# Optimal keep-alive window: the idle timeout minimizing idle GPU cost plus expected cold-start penalty
import math

def optimal_idle_timeout_s(
    gpu_cost_per_s: float,
    cold_start_penalty_cost: float,
    mean_interarrival_s: float,
) -> float:
    # Models request arrivals as a Poisson process with rate 1 / mean_interarrival_s.
    # Keeping a replica warm for t more seconds costs gpu_cost_per_s * t, and saves
    # an expected cold_start_penalty_cost * P(next request arrives within t).
    # The breakeven point sets marginal idle cost equal to marginal expected savings:
    #   gpu_cost_per_s == cold_start_penalty_cost * lambda * exp(-lambda * t)
    lam = 1.0 / mean_interarrival_s
    ratio = gpu_cost_per_s / (cold_start_penalty_cost * lam)
    if ratio >= 1:
        return 0.0  # idle cost already exceeds expected savings at t=0, scale down immediately
    return -math.log(ratio) / lam

Scaling and Performance

A single replica’s memory ceiling does not scale, since one A100 holds exactly one 26GB copy of the model plus whatever KV cache fits in the remainder. What scales is the number of independent replicas, and the number that needs to exist at any moment is driven directly by the gap between baseline traffic and burst traffic.

Given:
  - 13B params, FP16 = 26GB weights, A100 40GB per replica
  - KV cache per token = 2 * 40 layers * 8 kv heads * 128 head_dim * 2 bytes = 163,840 bytes (160 KB)
  - Headroom after weights + ~3GB activation/CUDA overhead = 40 - 26 - 3 = 11GB = 11,534,336 KB
  - Token capacity per replica: 11,534,336 KB / 160 KB = 72,089 tokens
  - Avg context length 900 tokens -> concurrent sequences per replica = 72,089 / 900 = ~80
  - Avg total occupancy per request (prefill + decode) = 7.0s

Baseline: 20 req/s * 7.0s = 140 concurrent sequences in flight
Baseline replicas: ceil(140 / (80 * 0.75)) = ceil(140 / 60) = 3

Burst: 200 req/s * 7.0s = 1,400 concurrent sequences in flight
Burst replicas: ceil(1,400 / 60) = 24

Gap to cover during a burst: 24 - 3 = 21 replicas, either pre-warmed ahead of
the spike by the predictor's 15s lead time, or absorbed by the admission
queue across parallel 3.6s snapshot restores

The concurrency ceiling per replica is set entirely by KV cache headroom, since the weights are fixed at 26GB no matter how many sequences are active. Caching strategy here is almost entirely write-once, read-many: a snapshot is written once per model version after its first cold boot, and read on every subsequent restore, so the NVMe cache’s hot spot is a handful of frequently restored models, not a constantly rewritten working set.

Cost and Token Economics

The dollar case for this design comes almost entirely from how small the baseline warm pool can be, not from any particular cleverness in the restore path itself. A GPU costs the same per hour whether it is serving traffic or sitting idle, so the question is how few idle GPU-hours a given SLA actually requires.

Configuration	GPU-hours/day	Cost/day at $2.80/GPU-hr	Cost / 1,000 requests	SLA compliance
Always-on, sized for peak (24 replicas, 24h)	576	$1,613	$0.93	~100%, but 21 GPUs idle most hours
Scale-to-zero, no snapshot (forced to keep ~20 warm to avoid 60-90s cold boots)	480	$1,344	$0.78	~100%, barely cheaper than always-on
Scale-to-zero + snapshot restore (reactive, 3 warm baseline)	114	$319	$0.18	~97%, occasional queue timeouts on unforecasted spikes
Scale-to-zero + snapshot + predictive pre-warm (chosen)	126	$353	$0.20	~99.5%+

The measured optimization is the move from “scale-to-zero, no snapshot” to “scale-to-zero with snapshot restore”: cost per 1,000 requests drops from $0.78 to $0.18, roughly a 77% reduction, purely because a 3.6 second restore lets the baseline warm pool shrink from near-peak size down to a small buffer. Adding the predictor on top costs a small premium, about two cents per 1,000 requests, in exchange for closing most of the remaining SLA gap.

Quality, Evaluation, and Guardrails

The quality question here is not whether the model’s outputs are good, the weights never change, it is whether a restored replica behaves identically to one that booted the ordinary way. Because there is no model quality to re-evaluate, the eval bar is a correctness diff, not a benchmark score.

# Restore correctness gate: blocks a snapshot from being marked ready if its output
# diverges from a natively warm replica on a fixed golden prompt set and seed
def evaluate_restored_replica(golden_tokens_reference: list[int], golden_tokens_restored: list[int]) -> dict:
    mismatches = sum(1 for a, b in zip(golden_tokens_reference, golden_tokens_restored) if a != b)
    total = len(golden_tokens_reference)
    return {
        "mismatch_count": mismatches,
        "total_tokens": total,
        "pass": mismatches == 0,
    }

Offline, every new snapshot runs against a fixed golden prompt set with a fixed sampling seed before it is marked ready in the manifest table, comparing its output token-for-token against a natively warm replica’s output for the same prompts. Online, the guardrail on the request path is simpler: a driver version mismatch or a failed checksum blocks the restore outright and falls back to a full cold boot rather than risk resuming into a state nobody has verified. The metric that gates a rollout is restore correctness pass rate, and the threshold is zero tolerance, a single token mismatch fails the snapshot.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Driver or CUDA version skew between snapshot and node	manifest driver version check fails before restore	restore refused	fall back to full cold boot, requeue request, flag node pool for driver alignment
Snapshot corruption on NVMe	sha256 mismatch on read	restore fails	fetch known-good snapshot from regional object store, or trigger a fresh cold boot and re-snapshot
Predictor false negative on an unforecasted spike	admission queue depth crosses its alert threshold faster than restores can drain it	requests queue past the SLA wait bound	shed load with a 429 and retry-after, emergency-scale additional cold boots in parallel
Thundering herd of restores during a flash spike	concurrent restore requests exceed available idle GPU capacity in the pool	some restores queue behind others	admission controller caps concurrent restores, queues the rest, prioritizes by nearest deadline
NVMe cache eviction of a trending model’s snapshot	cache miss on restore attempt for a model expected to be cached	restore falls back to a slower regional object-store fetch	trend-weighted eviction policy, and re-stage the snapshot locally before eviction pressure recurs
Idle-timeout race, scale-down before snapshot write completes	scale-down request finds no ready snapshot row	next cold start for that model has nothing to restore from	scheduler blocks scale-down until the snapshot status is ready, or extends the idle timeout by the write duration

Comparison of Approaches

Approach	TTFT (cold path)	Cost / 1,000 req	Complexity	Failure mode	Best fit
Always-on dedicated GPU	N/A, no cold path	$0.93	Low	None from cold starts; pure cost waste during idle hours	Latency-critical workloads with steady, predictable traffic
Scale-to-zero, naive cold boot	60,000-90,000ms	$0.78 (must over-provision warm pool to hit SLA)	Low	Every genuine scale-to-zero event breaches any reasonable TTFT SLA	Batch or offline workloads with no interactive latency requirement
Scale-to-zero + local weight cache only	15,000-25,000ms	$0.35	Medium	Still pays full CUDA init and allocator warmup on every cold start	Workloads that can tolerate seconds-to-tens-of-seconds latency
Scale-to-zero + GPU memory snapshot restore	~3,600ms	$0.18	High	A version-mismatched or corrupted snapshot fails closed to a full cold boot	Interactive workloads needing single-digit-second worst case TTFT
Scale-to-zero + snapshot + predictive pre-warm (chosen)	~3,600ms, rare	$0.20	High	Predictor false negatives fall back to the same-cost snapshot path, not naive cold boot	Bursty, interactive production traffic with a hard TTFT SLA

For a production LLM endpoint with a real TTFT SLA, the combination of snapshot restore and predictive pre-warming is the right default: it gets the worst case down from a minute and a half to under four seconds, and it gets the common case down to a warm hit most of the time, all for a small premium over the snapshot-only baseline. Teams with genuinely steady, non-bursty traffic and no cost pressure can reasonably skip the predictor and run the snapshot path reactively, since the incremental complexity of prediction buys the least when demand barely varies.

Key Takeaways

• GPU memory snapshotting captures a fully warmed process, including its CUDA context and resident weights, so a cold start becomes a restore instead of a re-derivation.
• Predictive pre-warming acts on a traffic forecast’s leading indicators, not a lagging threshold, because the lead time has to exceed restore latency plus decision latency or the forecast arrives too late to matter.
• Warm pool sizing is an economic breakeven calculation between idle GPU cost and expected cold-start penalty, not a constant chosen once at launch and forgotten.
• Scale-to-zero without a fast restore path does not actually save much money, since hitting a real SLA forces the warm pool back up close to peak size anyway.
• Lazy, layer-streaming weight loading matters only for the rare truly-first cold boot of a new model version; every subsequent cold start for that model should use the snapshot path instead.
• Bounded admission queues with deadline-aware eviction absorb the gap between a burst arriving and a replica becoming ready, without letting one stuck restore pile up unbounded requests behind it.
• Driver version pinning on every snapshot manifest is what keeps a restore failing closed instead of resuming into silently wrong state.
• Scale-in must wait on snapshot durability, not just an idle timer, or the system quietly throws away the one artifact that makes the next cold start fast.

The counter-intuitive lesson is that scale-to-zero and low latency are not actually in tension once the expensive part of a cold start, moving and initializing 26GB of state, is decoupled from the request path entirely. The real design skill is not choosing between cheap and fast, it is recognizing that the boot sequence’s expensive steps only need to happen once per model version, ever, and every cold start after that should be a restore of already-done work.

Frequently Asked Questions

Q: Why not just always keep at least one replica warm per model and skip snapshotting entirely? A: A single always-warm replica handles low, steady traffic fine, but it does nothing for the gap between baseline and a 10x burst, which still needs new replicas to come online fast. Without a snapshot, each of those new replicas pays the full 60 to 90 second cold boot, so the always-warm baseline just delays the problem rather than solving it.

Q: Why not cache the weights on local NVMe and skip the full GPU memory snapshot? A: Caching weights removes the object storage fetch penalty, roughly 50 seconds of the naive cold start, but it does nothing for CUDA driver initialization, allocator warmup, or the first forward pass that primes kernel caches. In practice that leaves 15 to 25 seconds of cold start unaddressed, which still breaches a 4-second SLA.

Q: What is the actual dollar cost of running snapshot restore and predictive pre-warming versus naive scale-to-zero? A: At $2.80 per GPU-hour, naive scale-to-zero forced into keeping a near-peak warm pool costs about $0.78 per 1,000 requests. The snapshot and predictor combination costs about $0.20 per 1,000 requests, roughly a 74% reduction, because the baseline warm pool can be sized for typical load instead of peak load.

Q: How do you know a restored replica has not silently diverged from a natively warm one? A: Every new snapshot runs against a fixed golden prompt set with a fixed sampling seed before it is marked ready, comparing output token-for-token against a natively warm replica. Any mismatch fails the snapshot and it never gets used for a live restore.

Q: What happens the very first time a brand-new model version is deployed, with no snapshot to restore from? A: That cold boot pays close to the full naive cost, mitigated somewhat by the lazy, layer-streaming loader so the first layers can start computing before the entire 26GB has landed. Immediately after that first successful boot, the snapshot manager captures its state, and every subsequent cold start for that model version uses the fast restore path instead.

Q: Does this design change for a model too large to fit on a single GPU? A: Yes. A sharded, multi-GPU model’s snapshot spans multiple devices and an interconnect, and restoring it means bringing every shard back into a consistent state simultaneously rather than resuming one process. That is a meaningfully different problem, closer to the checkpoint and restart concerns of a sharded serving system than to this single-GPU cold-start design.

Interview Questions

Q: Walk through the full path of a request that arrives with no warm replica available, naming every component and the time cost at each step. Expected depth: candidate should trace router detecting no warm capacity, admission controller enqueuing with a deadline, snapshot manager fetching and verifying the snapshot, restore via GPUDirect Storage and CUDA context resume at roughly 3.6 seconds total, and the replica then serving the request like any warm one. Should be able to state which step dominates the time budget.

Q: How would you size the warm pool given a traffic distribution, and what tradeoff does that sizing actually balance? Expected depth: should derive replica count from Little’s Law, in-flight requests divided by per-replica concurrency capacity, and explain the balance between idle GPU-hour cost and the cost of a cold start breaching the SLA, ideally referencing a breakeven-style keep-alive calculation rather than a guessed constant.

Q: Why can’t you use CRIU alone to checkpoint a GPU-resident process, and what does a snapshot manager need instead? Expected depth: should explain that GPU device memory is managed by the CUDA driver outside the kernel’s normal process memory model, so checkpointing it requires an explicit driver-level hook to flush and later re-import device state, which standard CRIU does not provide on its own.

Q: How do you detect and recover from a corrupted or version-mismatched snapshot without ever serving incorrect output? Expected depth: should describe a manifest that pins driver version and a content checksum, both checked before restore, with any mismatch failing closed to a full cold boot rather than attempting a partial or best-effort restore.

Q: If your snapshot restore took 30 seconds instead of 3.6, how would your warm pool sizing and admission control change? Expected depth: should recognize that a 30 second restore requires a much larger baseline warm pool to keep the admission queue’s wait bound reasonable, that the predictor’s required lead time grows accordingly, and that the cost advantage over naive scale-to-zero shrinks substantially, changing the whole economic case for the design.