Imagine a hospital with two identical operating rooms side by side. When a procedure needs a new piece of equipment, the surgical team sets up the second room completely, runs a full checklist with dummy patients, then wheels the real patient through in one smooth handoff - with the old room standing by as a backup in case anything goes wrong. That is blue-green deployment: maintain two identical production environments, promote the new one when it’s healthy, and keep the old one hot enough to flip back to within seconds.

The challenge isn’t the concept - it’s the precision engineering underneath it. Traffic routing must be weight-based and instantaneous, not DNS-based and slow. The new environment must pass real health checks, not just “pod is running.” Existing user sessions can’t be dropped mid-flight. Databases shared by both environments must support both schema versions simultaneously. And the entire promotion sequence from 0% to 100% must complete in under 30 seconds with automatic rollback triggered the moment error rates cross a threshold.

A naive approach - stop the old version, start the new one - gives you zero downtime only in theory. In practice you get a deployment window where requests either queue (causing latency spikes) or fail (causing 503s). Even “rolling updates” in Kubernetes have a window where old and new versions coexist, with no clean boundary to roll back across. The real problem is that you need atomic traffic switching at the load balancer level, health-gated promotion, and a state machine that handles the messy middle states: what happens if green becomes unhealthy at 50% traffic? What if a database migration breaks backward compatibility? What if the health check itself is wrong?

We need to solve for sub-30-second full traffic switching, instant rollback with no traffic loss, graceful session draining, database compatibility across both environments, and a deployments state machine that survives partial failures and controller crashes. Let’s build it.

Requirements and Constraints

Functional Requirements

• Deploy a new application version (green) alongside the current version (blue) without any downtime
• Shift production traffic from blue to green progressively: starting at 0%, then canary (1-10%), then full (100%)
• Complete the full traffic shift in under 30 seconds once smoke tests pass
• Roll back to the previous environment within 5 seconds if error rates exceed a configurable threshold
• Run automated smoke tests against the green environment before any live traffic is sent
• Support graceful session draining - existing connections to blue complete before blue is decommissioned
• Persist deployment state so the controller can resume if it crashes mid-deployment
• Support multi-region deployments with per-region rollout sequencing

Non-Functional Requirements

• Traffic switching latency: weight changes propagate to all load balancer nodes within 2 seconds
• Rollback trigger time: from error rate breach to 0% green traffic in under 5 seconds
• Health check frequency: every 2 seconds during active deployment, every 30 seconds in steady state
• Availability: 99.99% for the deployment controller itself (it’s in the blast radius of every deploy)
• Concurrent deployments: at most one active deployment per service at a time, globally enforced via distributed lock
• Session drain timeout: configurable, default 30 seconds before forceful termination of old connections

Constraints and Assumptions

• Services are stateless at the application tier; session state lives in Redis or a database, not in process memory
• Database migrations must be backward-compatible (expand-contract pattern); the system doesn’t manage DB migrations
• Blue environment is kept hot for at least 1 hour after promotion before teardown - instant rollback window
• The system targets Kubernetes-native or ECS-based infrastructure; bare-metal deployments are out of scope
• No blue-green across heterogeneous infrastructure (e.g., mixing Lambda and containers) in a single deployment

High-Level Architecture

The system has five major components working in concert. The Deployment Controller is the brain - a state machine that drives the entire lifecycle from provisioning to promotion or rollback. The Traffic Splitter sits at the load balancer layer (Nginx/Envoy) and applies weight-based routing rules in real time. The Health Checker continuously probes both environments and feeds signals back to the controller. The Metrics Store holds error rate, latency, and SLO data that the rollback decision algorithm reads. The Deployment State Store (etcd or Postgres) provides distributed locking and durable state so controller restarts don’t lose progress.

Traffic enters through the DNS layer or a global load balancer that routes to a regional Traffic Splitter. The splitter holds upstream definitions for both blue and green backends, with weight values updated dynamically by the controller. During steady state, green weight is 0 - all traffic hits blue. When a deployment starts, the controller provisions green, waits for health checks to stabilize, runs smoke tests, then begins incrementing green’s weight in steps. The Health Checker sends synthetic probes every 2 seconds and the Metrics Store scrapes real traffic error rates from both upstreams. If either signal goes red during a shift, the controller flips back to 100% blue in a single atomic weight update.

The Deployment Controller

The Deployment Controller is the central authority for every state transition in a deployment. Think of it as an air traffic controller: it has the full picture of where every flight (environment) is, enforces sequencing, and can divert a flight (roll back) the moment a runway issue is detected.