Certificate Expired at Midnight


cloud-infrastructure devops security

System Design Scenario

Certificate Expired at Midnight

When automation works perfectly on the wrong machine

⏱ 12 min read 📐 Intermediate 🔒 Cloud Infrastructure

Friday 12:03 AM. David’s phone explodes with customer support tickets. “Site shows security warning.” “Can’t complete checkout.” “Says connection is not secure.” The monitoring shows green across the board - servers healthy, database responding, CDN cache hit rate at 94%. But 50,000 users are seeing browser warnings about an insecure connection.

SSL certificates are like driver’s licenses - they expire on a fixed schedule, and when they expire, everything stops working instantly. David checks the certificate management dashboard: the cert was auto-renewed successfully at 11:47 PM, thirteen minutes before expiration. The automation worked perfectly. The certificate is valid, signed by a trusted CA, and has another 90 days before the next renewal.

The problem isn’t the certificate. It’s the location. The certificate was renewed on the staging server, not the production load balancer. The automation ran exactly as designed, but it ran on the wrong machine. Production is still serving yesterday’s expired certificate while the valid certificate sits unused on a server that handles 200 requests per day instead of 200,000.

This is certificate lifecycle management failure. When your automation doesn’t account for the distributed nature of modern infrastructure, perfect execution becomes perfectly useless.

Why This Happens

The instinct is to set up certificate auto-renewal on individual servers - it’s the simplest path from manual certificate management to automation. Tools like certbot make this straightforward: install on a server, configure a cron job, and certificates renew themselves every 60 days.

But modern applications aren’t single servers. They’re distributed across load balancers, CDNs, multiple regions, and various cloud services. Each component that terminates TLS connections needs a copy of the certificate, and they all need to be updated when the certificate renews.

The failure chain looks like this:

cert expires at midnight
  -> auto-renewal runs on server-1
    -> new cert generated successfully  
      -> server-1 gets updated cert
        -> server-2, server-3, load-balancer still have old cert
          -> users see security warnings
            -> revenue loss, support tickets, trust erosion

The automation solved the wrong problem. It prevented certificate expiration on one server while ignoring the distributed certificate deployment challenge.

Key Insight

Certificate renewal is a two-phase problem: generation (create a new valid certificate) and distribution (deploy it everywhere that needs it) - most automation only solves the first phase.

The Naive Solution (and where it breaks)

Most engineers first try to solve this by running the same renewal automation on every server. If server-1 can renew certificates automatically, why not replicate that process on server-2, server-3, and the load balancer?

This approach is like having multiple people independently renewing the same driver’s license - it creates more problems than it solves.

Premium Content

Unlock the full article along with everything else in the archive — all in one place.

In-depth analysis Expert insights Full archive access
Unlock Full Article