Think about how a large corporate campus issues employee badges. A central badge office verifies your identity once, prints a badge with an expiry date, and every door reader on campus trusts that badge because it trusts the office that printed it, not because it re-checks your ID at every door. When your badge is about to expire, the office reissues one before the old one stops working. When you leave the company, the office can deactivate your badge, and every door reader needs to find out about that fast. An internal certificate authority is that badge office, except the badges are TLS certificates, the doors are mTLS handshakes between microservices, and there are 10,000 employees who all need a new badge every single week.

The hard part is not printing one certificate. Any engineer can run openssl req and get a working leaf certificate in a minute. The hard part is doing that continuously, correctly, for 10,000 independently deployed services, such that every one of them always holds a certificate that is valid right now, without a human ever running that command, and without a renewal ever requiring a service restart. A service that finds its certificate expired at 3am is a service that is down at 3am, and at this scale, some certificate somewhere is always close to its expiry.

A naive approach hands each team a long-lived certificate (a year, two years) and calls it done. That fails in two directions at once. Long-lived certificates mean a compromised private key stays useful for a long time, so the blast radius of any leak is large. And a fleet of certificates issued at different times with different lifetimes turns “which certs expire this month” into a spreadsheet nobody maintains, until one does and takes down a service that had been quietly running the same cert since a laptop demo eighteen months ago. Shortening certificate lifetimes fixes the blast radius problem but multiplies the operational problem: if renewal is manual, a 7-day certificate needs manual attention 50 times more often than a year-long one. The only way short-lived certificates work is if issuance and renewal are fully automated and the running service never needs a human, or even a restart, to pick up the new one.

We need to solve for three things simultaneously: an issuance flow that can prove a service is who it claims to be before handing it a certificate, a renewal system that runs continuously and finishes well before any certificate’s expiry with margin for failure, and a delivery mechanism that gets a new certificate into a live TLS listener without dropping the connections already using the old one.

Requirements and Constraints

Functional Requirements

• Issue leaf TLS certificates to any of 10,000 registered services through an ACME-compatible flow, validating the requesting workload’s identity before signing anything
• Automatically renew every issued certificate well before its expiry, with no manual step in the common case
• Deliver a renewed certificate into a running service’s active TLS listener and hot-swap it with zero dropped connections and no process restart
• Support certificate revocation for a compromised private key or a decommissioned service, exposed through both an OCSP responder and a periodically published CRL
• Enforce mTLS between services: verify not just that a peer’s certificate is valid, but that its identity matches what the caller expects before allowing the connection
• Maintain a complete audit trail of every certificate issued, renewed, and revoked, including which service requested it, which validation method was used, and who or what approved it
• Support a CA hierarchy with an offline root and one or more online intermediates, so the root’s signing key can be rotated without reissuing every leaf certificate in the fleet

Non-Functional Requirements

• Issuance latency: p99 under 400ms for a single CSR-to-signed-certificate round trip through the issuance service
• Throughput: sustained 50 issuances/second in steady state, bursting to 500 issuances/second for mass-renewal events or a new environment bootstrapping 10,000 services at once
• Certificate lifetime: 7 days by default for leaf certificates, with renewal triggered at roughly 60% of lifetime elapsed
• Hot reload guarantee: swapping a certificate into a live listener must not drop or reset a single in-flight TLS connection
• Revocation propagation: OCSP responses reflect a revocation within 60 seconds of the revoke call; the CRL refreshes every 15 minutes
• Availability: the issuance service targets 99.95%; the offline root CA has no availability target by design, since it is deliberately not reachable during normal operation
• Scale: 10,000 services, roughly 15,000 active leaf certificates once multi-SAN and dual-identity services are counted, spread across 5 regions

Constraints and Assumptions

• This is an internal PKI, not a publicly-trusted CA. Services trust the internal root because it is provisioned into their trust store at boot time, not because a public browser trusts it
• We assume every host already has some form of workload identity available at provisioning time (a cloud instance identity document, a SPIFFE/SPIRE agent, or a bootstrap token issued by the deployment pipeline). We treat that identity source as a black-box input rather than designing it from scratch
• We do not cover the physical security or procurement of the hardware security module (HSM) protecting the root key beyond noting that root key custody exists and is deliberately out of the software system’s control path
• We assume the network is otherwise untrusted internally as well as externally; this system is designed for a zero-trust internal network, not a perimeter-trusted one
• We do not implement a publicly-trusted ACME integration (no Let’s Encrypt certs for external-facing endpoints); this system is scoped to internal service-to-service and service-to-infrastructure TLS

High-Level Architecture

The system has six major components: the Offline Root CA, the Online Intermediate CA, the ACME Issuance Service, the Cert Agent running alongside every service, the Expiry Monitor and Renewal Scheduler, and the Revocation Service exposing OCSP and CRL.

The Offline Root CA exists to sign exactly one kind of thing: intermediate CA certificates. It lives in cold storage, behind an HSM, disconnected from any network, and is deliberately brought online only for a scheduled intermediate signing ceremony, maybe once every few years. The Online Intermediate CA is the workhorse: it holds a key in a KMS or HSM that is reachable over the network, and it is the thing that actually signs the 10,000-plus leaf certificates the fleet needs. The ACME Issuance Service sits in front of the intermediate and implements the client-facing protocol: it receives certificate signing requests (CSRs), runs an identity validation challenge against the requester, and if that passes, forwards the request to the intermediate CA for signing.

The Cert Agent is a small daemon or sidecar running next to every one of the 10,000 services. It generates the private key locally (the key never leaves the host it was generated on), submits the CSR to the ACME Issuance Service, receives the signed certificate, and atomically swaps it into the service’s live TLS configuration. The Expiry Monitor and Renewal Scheduler tracks every certificate’s expiry timestamp and proactively triggers renewal at the right time, spread out to avoid every Cert Agent trying to renew simultaneously. The Revocation Service is the fleet-wide answer to “is this certificate still good,” available both as a fast OCSP lookup for a single certificate and as a downloadable CRL for verifiers that prefer to check locally.

A certificate’s journey looks like this: a service starts up, its Cert Agent generates a keypair and a CSR, and submits it to the ACME Issuance Service along with proof of the workload’s identity. The issuance service validates that proof against the service registry, forwards the CSR to the Intermediate CA, gets back a signed leaf certificate, and returns it to the Cert Agent. The Cert Agent hot-loads it into the running service’s TLS listener. From then on, the Expiry Monitor watches that certificate’s clock, and at roughly 60% of its 7-day lifetime, it nudges the Cert Agent to repeat the whole flow, well before the old certificate becomes a problem. If a key is ever suspected compromised, the Revocation Service marks the certificate revoked, and every peer doing an mTLS handshake finds out within the propagation SLA, not at the certificate’s natural expiry.

Component Deep Dives

The CA Hierarchy: Root and Intermediate

This component’s job is to establish a chain of trust that every service can verify offline, while keeping the most valuable key in the system reachable by almost nothing.

A single flat CA, where one key both signs intermediates and signs every leaf certificate directly, has an ugly property: that one key is both extremely valuable (compromising it compromises every certificate ever issued) and extremely busy (it has to be online to service thousands of issuance requests a day). Those two properties should never live on the same key. A two-tier hierarchy separates them: the root signs a small number of long-lived intermediate certificates and then goes back into cold storage, while an intermediate does the daily work of signing leaves and can be rotated, revoked, or replaced without ever touching the root.

Think of the root like a country’s master key-cutting die, kept in a vault and used only to authorize regional locksmiths. The regional locksmith (the intermediate) is the one who actually cuts keys for individual buildings (leaf certificates) all day long. If a regional locksmith’s tools are stolen, you revoke that locksmith’s authorization and stand up a new one; you don’t melt down the master die.