The Region That Went Dark


cloud-infrastructure reliability

System Design Scenario

The Region That Went Dark

When your entire stack lives in one AWS region and that region decides to take the night off

⏱ 12 min read 📐 Intermediate 🔒 Cloud Infrastructure

It’s 2:14 AM on a Tuesday. Your phone buzzes with the particular vibration pattern that means PagerDuty, not a text from a friend. You fumble it off the nightstand. CRITICAL: All health checks failing. Error rate: 100%. Revenue impact: $47K/min. You open your laptop and try to hit the dashboard. Nothing loads. You try the API directly. Connection timeout. You check Twitter - it’s already trending: “AWS us-east-1 outage.”

Your entire company - every service, every database, every queue, every byte of customer data - lives in a single AWS region. And that region just went dark.

This isn’t a hypothetical. On November 25, 2020, AWS us-east-1 experienced a major outage that took down a significant portion of the internet. Slack, Roku, Adobe, Flickr, Coinbase - all offline. On December 7, 2021, it happened again. The teams that recovered in minutes had one thing in common: they weren’t depending on a single region. The teams that were down for hours had something else in common: they’d been meaning to fix that.

You open your infrastructure diagram. It’s a single box labeled “us-east-1” with everything inside it. Load balancer, app servers, RDS database, ElastiCache, SQS queues, S3 buckets. The architecture that was “good enough for now” six months ago is now costing the company $2,800 every minute you sit here in the dark.

This is the multi-region availability problem - and the gap between knowing you need it and actually implementing it is where outages live.

Why This Happens

The default path on every cloud provider leads to single-region deployments. When you spin up an EC2 instance, it goes in one region. When you create an RDS database, it lives in one region. When you follow the “getting started” tutorial, everything lands in us-east-1 because it’s the default, it has the most services available, and it’s where all the examples point.

Single-region architectures aren’t a mistake - they’re a gravitational inevitability. Every additional region doubles your infrastructure cost, adds data consistency complexity, and introduces failure modes that didn’t exist before. Teams avoid multi-region because the engineering cost is real and the outage probability feels theoretical. Until it isn’t.

The failure chain is predictable:

AWS region experiences control plane issue
  → EC2 instances become unreachable
    → ALB health checks fail, no healthy targets
      → DNS still points to dead load balancer
        → All requests timeout (no alternative target)
          → 100% of users experience outage
            → Recovery depends entirely on AWS fixing the issue
              → You have zero control over your RTO

The core issue isn’t that regions fail - it’s that a single-region architecture has zero redundancy at the highest level of the infrastructure hierarchy. You might have three Availability Zones within that region, giving you redundancy against a single data center failure. But AZs share control planes, networking fabric, and regional services. When the region goes, the AZs go with it.

Core Insight

Availability Zones protect you from localized hardware failures. They do not protect you from regional control plane outages, regional networking issues, or regional service degradations. Multi-AZ is not multi-region.

The Naive Solution

The first thing most teams reach for after a regional outage is active-passive failover - a cold standby region that sits idle until needed. The reasoning is straightforward: keep a copy of everything in us-west-2, and if us-east-1 goes down, flip traffic over.

Single region architecture showing complete failure when us-east-1 goes down

The problem with cold standby is that it’s cold. Your secondary region hasn’t served real traffic in months. The database replica exists, but has it been tested? Are the ECS task definitions up to date? Did someone remember to deploy last Thursday’s config change to the standby region? The AMIs might be three versions behind. The IAM roles might reference resources that no longer exist.

Here’s where active-passive breaks down at scale:

Premium Content

Unlock the full article along with everything else in the archive — all in one place.

In-depth analysis Expert insights Full archive access
Unlock Full Article