4:59 PM, Friday

Marcus is a senior engineer who has shipped hundreds of deployments. He is careful. He reviews his diffs. He runs the test suite locally. And today he has a genuinely small fix: a one-line change to the session validation logic that prevents a rare but annoying login bug reported by a handful of enterprise customers.

It is 4:59 PM. The team is mentally wrapping up. He hits deploy.

By 5:05 PM, the on-call phone rings.

Users across three time zones are getting logged out mid-session. Customer support tickets are flooding in. The error rate dashboard turns red. The CEO texts the CTO. The CTO texts Marcus.

The fix was one line. The incident took four hours to resolve, two hours to write up, and one very long Monday morning to explain to the leadership team. And the postmortem revealed a painful truth: it was not the code that failed. It was the deployment strategy.

---

What Actually Went Wrong

Marcus deployed using a standard rolling update: instances were killed and replaced with the new version one by one. Here is what that looked like from the outside:

Rolling Deploy - 8 instances

t=0:00  All running version 1 (stable)
        [v1][v1][v1][v1][v1][v1][v1][v1]

t=0:30  First instance replaced
        [v2][v1][v1][v1][v1][v1][v1][v1]
        12.5% of traffic now hitting v2

t=1:00  Second instance replaced
        [v2][v2][v1][v1][v1][v1][v1][v1]
        25% of users getting unexpected logouts

t=1:30  Third instance replaced
        [v2][v2][v2][v1][v1][v1][v1][v1]
        37% impacted - on-call fires

t=2:00  Deploy paused manually
        Three mixed-version minutes already done
        Rollback requires another full 8-minute deploy cycle

The failure chain:

1. The new session validation code was subtly incompatible with tokens issued by the old version
2. Users whose requests landed on a v2 instance had their sessions invalidated
3. The load balancer mixed v1 and v2 traffic randomly on every request
4. Users experienced random logouts depending on which instance they hit each time
5. Rolling back required another 8-minute deploy cycle of the same mixed-version problem
6. Total blast radius: 35% of active users, 13 minutes of degraded experience

The core problem: there was no safe way to validate the new version in production before it touched real users, and no instant escape hatch when it misbehaved. The team discovered the bug by watching user impact, not by catching it first.

---

Solution 1: Blue-Green Deployments