A highway department repaving a bridge deck does not close all six lanes overnight and gamble that the new asphalt holds under a full day of traffic. It opens one lane, watches it carry a small fraction of the cars first, and keeps a crew standing by who can throw up cones and route everyone back onto the old lanes within minutes if the new surface cracks. Shipping a new machine learning model into a system that scores 200,000 inference requests a second is the same problem wearing different clothes. You cannot cut every request over to a freshly trained model and hope its offline validation metrics translate cleanly to live traffic, because they often do not.

The closer analogy is a clinical drug trial. A new drug is never given to every patient the day it clears the lab. It is given to a randomly assigned fraction of patients while the rest continue on the existing standard of care, and a safety monitoring board watches the trial’s outcomes continuously, empowered to halt it the moment the new drug looks like it is doing more harm than good, well before the trial reaches its planned end date. A model versioning and A/B testing system is that trial infrastructure built into an inference pipeline: a model registry that tracks exactly which version is control and which is the challenger, a routing layer that assigns each user to one arm of the experiment and keeps them there, a metrics pipeline that watches the business outcome for each arm separately, and a monitoring board, implemented as code, that pulls the plug automatically when the challenger is losing.

The naive approach, deploy the new model behind a flag and eyeball a dashboard for a week, fails for reasons that only show up at scale. A model serving 200,000 requests a second across 500 registered versions and 50 concurrent experiments cannot be evaluated by a human staring at a graph, because the graph is noisy. A 0.3% dip in conversion rate over one hour is completely ordinary sampling variance for most products, and reacting to every dip means the challenger never gets a fair trial. Worse, if the same user sees the old model on one request and the new model on the next, no metric computed downstream means anything, because you can no longer attribute a purchase or a click to the model that actually produced the recommendation the user acted on. And if the team decides to roll back manually after spotting a problem in a dashboard, the decision typically lands hours after the damage was already done to tens of millions of served requests.

The forces in tension are speed against confidence, and isolation against realism. We need a router that assigns traffic instantly, adding no meaningful latency to the hot inference path, while guaranteeing that one user’s experience stays consistent for the entire life of an experiment. We need a metrics system that separates signal from noise fast enough to catch a bad model within minutes, not weeks, without falsely killing challengers that are actually fine. And we need a rollback mechanism that acts automatically, because deciding “this is really worse, not just noisy” is exactly the kind of statistical judgment call a computer can make faster and more consistently than an on-call engineer at 3 AM. We need to solve for traffic splitting that is deterministic and sticky, metric collection that is attributable to the exact model version that produced each prediction, and statistical decision-making that is both fast and resistant to false alarms, all at the same time.

Requirements and Constraints

Functional Requirements

• Register every trained model artifact in a model registry with a unique version, lineage (training run, parent version), and a promotion state (staging, shadow, canary, production, deprecated, archived)
• Split live inference traffic between two or more versions of the same model according to a configured percentage, routing each request deterministically to exactly one variant
• Guarantee that the same entity (typically a user) is assigned to the same variant for the full duration of an experiment, even across router restarts and rolling deploys (experiment assignment consistency)
• Run a challenger model in shadow mode: score production traffic against it without ever returning its prediction to the caller, purely to compare its predictions and latency against the currently served model before it earns any live traffic
• Collect business metrics, not just model accuracy, per model variant, tagged with the same experiment and variant identifiers used for routing (metric collection per variant)
• Continuously test whether the difference in a guardrail metric between variants is statistically significant, not just directionally different (statistical significance testing)
• Automatically reduce a challenger’s traffic allocation to zero when it breaches a guardrail metric with statistical confidence, without waiting for a human to intervene (automatic rollback triggers)
• Allow an operator to manually pause, override, or force-promote an experiment at any point

Non-Functional Requirements

• Scale: 200,000 inference requests/sec at peak across the serving fleet, 500 registered model versions, 50 concurrent experiments, roughly 20 million daily active entities receiving an assignment
• Routing latency: the assignment decision adds no more than 1ms at p99 on top of the model’s own inference latency
• Assignment consistency: 100% of repeat requests from the same entity within an experiment’s lifetime resolve to the same variant, continuously verified
• Metrics freshness: guardrail metrics roll up into 5-minute windows, available to the statistical engine within 60 seconds of window close
• Statistical rigor: a 0.01 significance threshold (99% confidence) on guardrail metrics before an automatic rollback fires, with a minimum sample size of 10,000 assigned entities per variant before any test is evaluated
• Rollback latency: under 3 minutes end to end, from a confirmed guardrail breach to challenger traffic reaching 0%
• Availability: 99.99% for the traffic router and serving path; the statistical analysis and rollback path can tolerate a few minutes of delay without materially changing the outcome
• Durability: every assignment, prediction event, and rollback decision is logged and retained for at least 180 days for audit and post-hoc analysis

Constraints and Assumptions

• We are not building the model training pipeline or the feature store that supplies inference inputs; this design assumes trained model artifacts and computed features already exist
• We are not building a general-purpose statistics platform for arbitrary product experiments like pricing or UI copy; this focuses specifically on ML model rollout experiments, though the primitives generalize
• We assume model artifacts are already validated offline (held-out test set, bias checks) before entering the registry; this system governs the online rollout, not offline model quality
• We assume the inference serving infrastructure itself (the GPU/CPU fleet, autoscaling) already exists; this design covers the routing, measurement, and rollback layer sitting on top of it

High-Level Architecture

The system has six major components: the Model Registry, the Traffic Router, the Shadow Evaluation path, the Metrics Pipeline, the Statistical Analysis Engine, and the Rollback Controller.

A request carrying an entity identifier arrives at the Traffic Router, which first checks the Model Registry for the active experiment configuration on the requested model slot: which versions are eligible, and what split percentages apply. The router deterministically resolves the entity to a variant, forwards the request to the control model on the serving fleet for the overwhelming majority of traffic, and to the challenger model for the configured canary percentage. In parallel, and completely invisibly to the caller, a configured fraction of traffic is also mirrored to any model running in shadow mode, whose response is scored and logged but never returned. Every served and shadowed prediction, along with any business outcome that follows it, streams into the Metrics Pipeline, which rolls per-variant events into short aggregation windows. The Statistical Analysis Engine reads those windows continuously and tests whether the challenger’s guardrail metrics differ from control’s by more than chance. When that test confirms a real regression, the Rollback Controller reduces the challenger’s traffic allocation back to the router’s configuration, closing the loop without a human in it.

The Model Registry sits beside the request path rather than inside it. The router reads the registry’s cached configuration on every request, but the registry itself is a control-plane system, not a data-plane one; its writes happen when engineers register new model versions or when the Rollback Controller updates a split percentage, not on every inference call.

Component Deep Dives

The Model Registry

This component’s job is to be the definitive, versioned record of every model artifact allowed to receive live traffic, and the promotion state that governs whether it currently can.

Think of it like a hospital’s drug formulary. Nothing gets administered to a patient unless it is on the approved list, its exact batch is traceable back to a manufacturing run, and its current status, approved, restricted, or recalled, is checked before every dose. The registry plays the same role for a model version: model_version_id is the batch number, promotion_state is the formulary status, and no router, ever, is allowed to route traffic to a version by convention or by directory name instead of by an explicit registry lookup.

-- Model registry: every artifact, its lineage, and its current promotion state
CREATE TABLE model_versions (
    model_version_id      BIGSERIAL PRIMARY KEY,
    model_name              TEXT NOT NULL,
    version_number             INTEGER NOT NULL,
    artifact_uri                  TEXT NOT NULL,
    training_run_id                 TEXT NOT NULL,
    parent_version_id                 BIGINT REFERENCES model_versions(model_version_id),
    promotion_state                     TEXT NOT NULL DEFAULT 'staging'
        CHECK (promotion_state IN ('staging', 'shadow', 'canary', 'production', 'deprecated', 'archived')),
    created_at                             TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (model_name, version_number)
);

-- Only one production version per model slot, outside a brief migration window
CREATE UNIQUE INDEX one_production_per_model
    ON model_versions (model_name)
    WHERE promotion_state = 'production';

CREATE INDEX ON model_versions (model_name, promotion_state);

If you skip the registry and let the router simply point at whatever artifact happens to live in a path called latest, a training job that overwrites latest mid-experiment silently swaps out the challenger a live experiment is measuring. Every metric collected before and after that swap gets attributed to a single model_version_id in your dashboards, when it was actually produced by two different models, and there is no trace of when the swap happened. The registry closes that gap by making the artifact reference immutable once a model_version_id is created; a new artifact always means a new row, never an update to an existing one.