Think about how a large corporate campus issues employee badges. A central badge office verifies your identity once, prints a badge with an expiry date, and every door reader on campus trusts that badge because it trusts the office that printed it, not because it re-checks your ID at every door. When your badge is about to expire, the office reissues one before the old one stops working. When you leave the company, the office can deactivate your badge, and every door reader needs to find out about that fast. An internal certificate authority is that badge office, except the badges are TLS certificates, the doors are mTLS handshakes between microservices, and there are 10,000 employees who all need a new badge every single week.

The hard part is not printing one certificate. Any engineer can run openssl req and get a working leaf certificate in a minute. The hard part is doing that continuously, correctly, for 10,000 independently deployed services, such that every one of them always holds a certificate that is valid right now, without a human ever running that command, and without a renewal ever requiring a service restart. A service that finds its certificate expired at 3am is a service that is down at 3am, and at this scale, some certificate somewhere is always close to its expiry.

A naive approach hands each team a long-lived certificate (a year, two years) and calls it done. That fails in two directions at once. Long-lived certificates mean a compromised private key stays useful for a long time, so the blast radius of any leak is large. And a fleet of certificates issued at different times with different lifetimes turns “which certs expire this month” into a spreadsheet nobody maintains, until one does and takes down a service that had been quietly running the same cert since a laptop demo eighteen months ago. Shortening certificate lifetimes fixes the blast radius problem but multiplies the operational problem: if renewal is manual, a 7-day certificate needs manual attention 50 times more often than a year-long one. The only way short-lived certificates work is if issuance and renewal are fully automated and the running service never needs a human, or even a restart, to pick up the new one.

We need to solve for three things simultaneously: an issuance flow that can prove a service is who it claims to be before handing it a certificate, a renewal system that runs continuously and finishes well before any certificate’s expiry with margin for failure, and a delivery mechanism that gets a new certificate into a live TLS listener without dropping the connections already using the old one.

Requirements and Constraints

Functional Requirements

• Issue leaf TLS certificates to any of 10,000 registered services through an ACME-compatible flow, validating the requesting workload’s identity before signing anything
• Automatically renew every issued certificate well before its expiry, with no manual step in the common case
• Deliver a renewed certificate into a running service’s active TLS listener and hot-swap it with zero dropped connections and no process restart
• Support certificate revocation for a compromised private key or a decommissioned service, exposed through both an OCSP responder and a periodically published CRL
• Enforce mTLS between services: verify not just that a peer’s certificate is valid, but that its identity matches what the caller expects before allowing the connection
• Maintain a complete audit trail of every certificate issued, renewed, and revoked, including which service requested it, which validation method was used, and who or what approved it
• Support a CA hierarchy with an offline root and one or more online intermediates, so the root’s signing key can be rotated without reissuing every leaf certificate in the fleet

Non-Functional Requirements

• Issuance latency: p99 under 400ms for a single CSR-to-signed-certificate round trip through the issuance service
• Throughput: sustained 50 issuances/second in steady state, bursting to 500 issuances/second for mass-renewal events or a new environment bootstrapping 10,000 services at once
• Certificate lifetime: 7 days by default for leaf certificates, with renewal triggered at roughly 60% of lifetime elapsed
• Hot reload guarantee: swapping a certificate into a live listener must not drop or reset a single in-flight TLS connection
• Revocation propagation: OCSP responses reflect a revocation within 60 seconds of the revoke call; the CRL refreshes every 15 minutes
• Availability: the issuance service targets 99.95%; the offline root CA has no availability target by design, since it is deliberately not reachable during normal operation
• Scale: 10,000 services, roughly 15,000 active leaf certificates once multi-SAN and dual-identity services are counted, spread across 5 regions

Constraints and Assumptions

• This is an internal PKI, not a publicly-trusted CA. Services trust the internal root because it is provisioned into their trust store at boot time, not because a public browser trusts it
• We assume every host already has some form of workload identity available at provisioning time (a cloud instance identity document, a SPIFFE/SPIRE agent, or a bootstrap token issued by the deployment pipeline). We treat that identity source as a black-box input rather than designing it from scratch
• We do not cover the physical security or procurement of the hardware security module (HSM) protecting the root key beyond noting that root key custody exists and is deliberately out of the software system’s control path
• We assume the network is otherwise untrusted internally as well as externally; this system is designed for a zero-trust internal network, not a perimeter-trusted one
• We do not implement a publicly-trusted ACME integration (no Let’s Encrypt certs for external-facing endpoints); this system is scoped to internal service-to-service and service-to-infrastructure TLS

High-Level Architecture

The system has six major components: the Offline Root CA, the Online Intermediate CA, the ACME Issuance Service, the Cert Agent running alongside every service, the Expiry Monitor and Renewal Scheduler, and the Revocation Service exposing OCSP and CRL.

The Offline Root CA exists to sign exactly one kind of thing: intermediate CA certificates. It lives in cold storage, behind an HSM, disconnected from any network, and is deliberately brought online only for a scheduled intermediate signing ceremony, maybe once every few years. The Online Intermediate CA is the workhorse: it holds a key in a KMS or HSM that is reachable over the network, and it is the thing that actually signs the 10,000-plus leaf certificates the fleet needs. The ACME Issuance Service sits in front of the intermediate and implements the client-facing protocol: it receives certificate signing requests (CSRs), runs an identity validation challenge against the requester, and if that passes, forwards the request to the intermediate CA for signing.

The Cert Agent is a small daemon or sidecar running next to every one of the 10,000 services. It generates the private key locally (the key never leaves the host it was generated on), submits the CSR to the ACME Issuance Service, receives the signed certificate, and atomically swaps it into the service’s live TLS configuration. The Expiry Monitor and Renewal Scheduler tracks every certificate’s expiry timestamp and proactively triggers renewal at the right time, spread out to avoid every Cert Agent trying to renew simultaneously. The Revocation Service is the fleet-wide answer to “is this certificate still good,” available both as a fast OCSP lookup for a single certificate and as a downloadable CRL for verifiers that prefer to check locally.

A certificate’s journey looks like this: a service starts up, its Cert Agent generates a keypair and a CSR, and submits it to the ACME Issuance Service along with proof of the workload’s identity. The issuance service validates that proof against the service registry, forwards the CSR to the Intermediate CA, gets back a signed leaf certificate, and returns it to the Cert Agent. The Cert Agent hot-loads it into the running service’s TLS listener. From then on, the Expiry Monitor watches that certificate’s clock, and at roughly 60% of its 7-day lifetime, it nudges the Cert Agent to repeat the whole flow, well before the old certificate becomes a problem. If a key is ever suspected compromised, the Revocation Service marks the certificate revoked, and every peer doing an mTLS handshake finds out within the propagation SLA, not at the certificate’s natural expiry.

Component Deep Dives

The CA Hierarchy: Root and Intermediate

This component’s job is to establish a chain of trust that every service can verify offline, while keeping the most valuable key in the system reachable by almost nothing.

A single flat CA, where one key both signs intermediates and signs every leaf certificate directly, has an ugly property: that one key is both extremely valuable (compromising it compromises every certificate ever issued) and extremely busy (it has to be online to service thousands of issuance requests a day). Those two properties should never live on the same key. A two-tier hierarchy separates them: the root signs a small number of long-lived intermediate certificates and then goes back into cold storage, while an intermediate does the daily work of signing leaves and can be rotated, revoked, or replaced without ever touching the root.

Think of the root like a country’s master key-cutting die, kept in a vault and used only to authorize regional locksmiths. The regional locksmith (the intermediate) is the one who actually cuts keys for individual buildings (leaf certificates) all day long. If a regional locksmith’s tools are stolen, you revoke that locksmith’s authorization and stand up a new one; you don’t melt down the master die.

# Root CA generation - run once, offline, on an air-gapped machine
# Demonstrates: root key generation and self-signed root certificate
openssl genpkey -algorithm ED25519 -out root-ca.key
openssl req -x509 -new -key root-ca.key -sha256 -days 7300 \
  -subj "/O=InternalPKI/CN=InternalPKI Root CA G1" \
  -out root-ca.crt \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign"

# Intermediate CA CSR - generated online, signed offline by the root
openssl genpkey -algorithm ED25519 -out intermediate-ca.key
openssl req -new -key intermediate-ca.key \
  -subj "/O=InternalPKI/CN=InternalPKI Issuing CA us-east-1" \
  -out intermediate-ca.csr

# Signing ceremony - performed with the root key, witnessed, logged, key never leaves the HSM
openssl x509 -req -in intermediate-ca.csr -CA root-ca.crt -CAkey root-ca.key \
  -CAcreateserial -days 1825 -sha256 -out intermediate-ca.crt \
  -extfile <(printf "basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,keyCertSign,cRLSign")

The pathlen:1 on the root and pathlen:0 on the intermediate are not decoration. pathlen:0 on the intermediate certificate tells every verifier that this intermediate cannot sign further intermediates, only leaf certificates, closing off an entire class of privilege-escalation bug where a compromised intermediate mints its own sub-intermediate. Every region gets its own intermediate signed by the same root, so a compromise in us-east-1’s intermediate can be revoked without touching eu-west-1’s.

What breaks if you skip the hierarchy and let one key do everything: root key rotation becomes an all-or-nothing event. Rotating a single flat CA’s key means every service in the fleet needs a new trust anchor simultaneously, which at 10,000 services is a coordination problem with no good rollback. With a hierarchy, root rotation only requires re-signing the small number of intermediates, and services that trust the root by fingerprint rather than by a single hardcoded cert can pick up the new intermediate transparently through the chain.

The ACME Issuance Service

This component’s job is to turn “a service claims to be orders-api and wants a certificate” into a signed certificate only if that claim can actually be verified, using a protocol clients already know how to speak.

The obvious mistake here is treating issuance as “anyone who can reach this endpoint gets a cert.” At 10,000 services, the issuance service is effectively the front door to the entire trust fabric, and its authorization check is the single control standing between a compromised host and a valid identity certificate for any service it wants to impersonate.

We adopt ACME (RFC 8555), the protocol Let’s Encrypt popularized, because the client-side tooling and the order/challenge/finalize state machine are already well understood and have solid open-source client libraries. The public internet version of ACME proves domain ownership using HTTP-01 or DNS-01 challenges. We do not own arbitrary domains internally in the same way, so we implement a custom challenge type, attestation-01, that validates a workload identity token instead of a DNS record.

ACME order flow (RFC 8555 subset, custom attestation-01 challenge):

1. POST /acme/new-order        {"identifiers": [{"type": "spiffe", "value": "spiffe://internal/ns/payments/sa/orders-api"}]}
   -> 201 {"status": "pending", "authorizations": ["/acme/authz/8f2c"]}

2. GET  /acme/authz/8f2c
   -> 200 {"status": "pending", "challenges": [{"type": "attestation-01", "url": "/acme/chall/91ab", "token": "d7f3..."}]}

3. Cert Agent presents its workload attestation document (signed by the cloud
   provider or SPIRE agent) alongside the challenge token to prove it is
   really running as orders-api, not just claiming to be.

   POST /acme/chall/91ab   {"attestation": "<signed-jwt>", "token": "d7f3..."}
   -> 200 {"status": "valid"}

4. POST /acme/finalize   {"csr": "<base64-der-csr>"}
   -> 200 {"status": "valid", "certificate": "/acme/cert/7e21"}

5. GET  /acme/cert/7e21
   -> 200 <PEM certificate chain: leaf + intermediate>

# ACME-style attestation-01 challenge validator
# Demonstrates: verifying a signed workload identity token against the
# service registry before authorizing issuance of a given SPIFFE identity
import time
import jwt  # PyJWT

TRUSTED_ATTESTOR_KEYS = load_attestor_public_keys()  # cloud provider / SPIRE agent keys

def validate_attestation_challenge(requested_identity: str, attestation_jwt: str, registry) -> bool:
    try:
        claims = jwt.decode(
            attestation_jwt,
            key=TRUSTED_ATTESTOR_KEYS,
            algorithms=["ES256"],
            options={"require": ["exp", "iat", "sub"]},
        )
    except jwt.InvalidTokenError:
        return False

    # The attestation token must be fresh - stale tokens could be replayed
    # from a host that has since been decommissioned or reimaged.
    if time.time() - claims["iat"] > 30:
        return False

    attested_workload = claims["sub"]  # e.g. "i-0abc123/orders-api"
    registered = registry.lookup_by_workload(attested_workload)
    if registered is None:
        return False

    # The identity the CSR is requesting must exactly match what the
    # attestation proves this specific host is allowed to run.
    return registered.allowed_identity == requested_identity

The CSR’s subject alternative name is not trusted at face value; it is cross-checked against the service registry entry tied to the attestation. A compromised host attempting to request a certificate for payments-api while its attestation proves it is only orders-api fails at this check, before the request ever reaches the intermediate CA’s signing key.

The Cert Agent and Hot Reload

This component’s job is to make a certificate renewal invisible to the service’s traffic, which means the private key must be generated locally, and the swap into the live listener must be atomic.

The naive way to “hot reload” a certificate is to write the new cert and key to disk and restart the process, or send it a SIGHUP and hope the framework re-reads the files cleanly. Both approaches either drop connections outright (a restart) or depend on every framework in a polyglot fleet implementing signal-based reload correctly, which in practice they mostly do not. The design we actually want swaps the certificate in memory, inside the running process, without tearing down the listener socket at all.

Go’s tls.Config supports this natively through GetCertificate, a callback invoked on every new TLS handshake rather than a certificate baked in at listener-start time. The Cert Agent updates an atomic pointer; every new incoming connection picks up the latest certificate, while every connection already in flight keeps using whichever certificate it started its handshake with, since TLS certificates are pinned per-connection, not per-listener.

// Hot-reloadable TLS certificate store
// Demonstrates: atomic pointer swap so in-flight connections are unaffected
// and every new handshake picks up the latest certificate with zero downtime
package certagent

import (
	"crypto/tls"
	"sync/atomic"
)

type CertStore struct {
	current atomic.Pointer[tls.Certificate]
}

func NewCertStore(initial tls.Certificate) *CertStore {
	cs := &CertStore{}
	cs.current.Store(&initial)
	return cs
}

// GetCertificate is wired into tls.Config.GetCertificate. It runs once per
// handshake, so swapping cs.current never touches sockets already established.
func (cs *CertStore) GetCertificate(_ *tls.ClientHelloInfo) (*tls.Certificate, error) {
	return cs.current.Load(), nil
}

// Swap is called by the renewal path once a new certificate has been
// validated. It never blocks a concurrent GetCertificate call.
func (cs *CertStore) Swap(next tls.Certificate) {
	cs.current.Store(&next)
}

// Renewal-triggered reload path
// Demonstrates: validating the new cert against the private key and expiry
// before ever exposing it to GetCertificate, so a corrupt renewal can't
// take the listener down
func (a *CertAgent) applyRenewedCertificate(certPEM, keyPEM []byte) error {
	pair, err := tls.X509KeyPair(certPEM, keyPEM)
	if err != nil {
		return fmt.Errorf("renewed cert failed to parse: %w", err)
	}
	leaf, err := x509.ParseCertificate(pair.Certificate[0])
	if err != nil {
		return fmt.Errorf("renewed cert failed to decode: %w", err)
	}
	if time.Until(leaf.NotAfter) < 24*time.Hour {
		return fmt.Errorf("renewed cert expires too soon, refusing to activate: %s", leaf.NotAfter)
	}
	pair.Leaf = leaf
	a.store.Swap(pair)
	a.metrics.RecordRotation(leaf.SerialNumber.String(), leaf.NotAfter)
	return nil
}

The analogy here is a relay runner handing off a baton without either runner stopping: the old certificate keeps serving every handshake that already grabbed it, the new certificate serves every handshake from the handoff point forward, and there is no moment where the track is empty. What breaks without this design: teams that reload by restarting the process end up scheduling renewals during low-traffic windows to minimize impact, which reintroduces exactly the kind of manual operational coordination automated rotation was supposed to eliminate.

Expiry Monitoring and Renewal Scheduling

This component’s job is to make sure a renewal always happens with margin to spare, and that 10,000 services do not all try to renew in the same ten-second window.

The obviously wrong approach is “renew at 99% of lifetime, right before expiry.” That leaves zero room for a transient failure: if the ACME Issuance Service is having a bad minute exactly when a certificate’s clock runs out, the service goes down with an expired cert and no time left to retry. The fix is to renew early, with jitter, and to keep retrying with backoff if the first attempt fails, all while still comfortably inside the validity window.

# Renewal scheduling: target renewal time with jitter to avoid thundering herd
# Demonstrates: proactive early renewal with randomized spread and bounded retry
import random
from datetime import datetime, timedelta, timezone

RENEWAL_THRESHOLD = 0.60   # renew once 60% of lifetime has elapsed
JITTER_FRACTION = 0.10     # +/- 10% of lifetime, spread across the fleet
MAX_RETRY_ATTEMPTS = 5
RETRY_BACKOFF_BASE_SECONDS = 30

def compute_renewal_time(issued_at: datetime, not_after: datetime) -> datetime:
    lifetime = not_after - issued_at
    base_offset = lifetime * RENEWAL_THRESHOLD
    jitter_window = lifetime.total_seconds() * JITTER_FRACTION
    jitter = random.uniform(-jitter_window, jitter_window)
    return issued_at + base_offset + timedelta(seconds=jitter)

def renew_with_backoff(cert_agent, attempt: int = 0) -> bool:
    try:
        cert_agent.request_renewal()
        return True
    except IssuanceServiceUnavailable:
        if attempt >= MAX_RETRY_ATTEMPTS:
            cert_agent.alert_operators("renewal exhausted retries, escalating")
            return False
        delay = RETRY_BACKOFF_BASE_SECONDS * (2 ** attempt) + random.uniform(0, 5)
        cert_agent.schedule_retry(delay_seconds=delay)
        return False

For a 7-day certificate, a 60% threshold means renewal is first attempted around day 4.2, with jitter spreading the actual attempt anywhere between roughly day 3.5 and day 4.9. That leaves more than two full days of runway even if the first several retry attempts fail, which is a wide enough window to absorb an issuance service outage that lasts hours, not just seconds.

The Expiry Monitor runs independently of the Cert Agents as a backstop: it periodically scans the certificate metadata table for anything approaching expiry without a corresponding recent renewal, and pages a human if a certificate is inside, say, 24 hours of expiry with no successful renewal on record. This catches the case a purely client-driven scheduler cannot: a Cert Agent that crashed, was never restarted, and stopped renewing anything at all.

OCSP and CRL: Revocation at Scale

This component’s job is to answer “is this specific certificate still trustworthy” fast enough to sit on the connection-setup path, for the rare case where a certificate needs to die before its natural expiry.

Short-lived certificates (7 days) already do most of the work of limiting how long a compromised key stays dangerous, which is deliberate: it means revocation infrastructure only has to handle the minority of cases where waiting out a natural expiry is not acceptable, like a confirmed key leak or an emergency service decommission. But that minority still needs a fast, reliable answer.

OCSP (Online Certificate Status Protocol) answers a single question for a single certificate, quickly: revoked, good, or unknown, keyed by the certificate’s serial number. A CRL (Certificate Revocation List) is the opposite tradeoff: a full, periodically republished list of every revoked-but-not-yet-expired certificate, meant for verifiers that would rather cache a whole list locally than make a network call per handshake.

# OCSP responder core logic
# Demonstrates: serial-number lookup against the revocation table, signed response
from datetime import datetime, timedelta, timezone

def build_ocsp_response(serial_number: str, revocation_db, responder_key) -> "OCSPResponse":
    entry = revocation_db.lookup(serial_number)
    now = datetime.now(timezone.utc)

    if entry is None:
        status = OCSPCertStatus.GOOD
    elif entry.revoked_at is not None:
        status = OCSPCertStatus.REVOKED
    else:
        status = OCSPCertStatus.GOOD

    response = OCSPResponse(
        serial_number=serial_number,
        cert_status=status,
        this_update=now,
        next_update=now + timedelta(minutes=10),  # short validity keeps stale caching bounded
        revocation_time=entry.revoked_at if entry else None,
        revocation_reason=entry.reason if entry else None,
    )
    return response.sign(responder_key)

-- Revocation table backing both the OCSP responder and the CRL generator
-- Demonstrates: fast serial-number lookup, indexed for the OCSP hot path
CREATE TABLE revoked_certificates (
    serial_number   BYTEA       PRIMARY KEY,
    issuer_key_hash BYTEA       NOT NULL,
    revoked_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    reason          TEXT        NOT NULL
        CHECK (reason IN ('key_compromise', 'superseded', 'cessation_of_operation', 'unspecified')),
    revoked_by      TEXT        NOT NULL   -- operator identity or automation source
);

CREATE INDEX ON revoked_certificates (issuer_key_hash, revoked_at);

OCSP responses are signed and short-lived (a 10-minute next_update above), which bounds how stale a cached “good” response can be without requiring every verifier to hit the responder on every single handshake; most TLS stacks staple a recently fetched OCSP response to the handshake itself (OCSP stapling) so the server, not each client, pays the lookup cost. The CRL is regenerated every 15 minutes and published to a well-known, heavily cached location, since it changes rarely and only grows when something is actually revoked, not on every renewal.

Data Model

The data model spans four entities: the certificate registry, the service identity registry, the revocation table (shown above), and the issuance audit log.

-- Certificate registry: the single source of truth for every issued leaf certificate
-- Demonstrates: expiry-indexed lookups for the renewal scheduler and expiry monitor
CREATE TABLE certificates (
    serial_number     BYTEA       PRIMARY KEY,
    service_id        TEXT        NOT NULL REFERENCES services(service_id),
    subject_identity  TEXT        NOT NULL,   -- e.g. spiffe://internal/ns/payments/sa/orders-api
    issuer_key_hash   BYTEA       NOT NULL,   -- which intermediate signed this
    public_key_hash   BYTEA       NOT NULL,
    issued_at         TIMESTAMPTZ NOT NULL,
    not_before        TIMESTAMPTZ NOT NULL,
    not_after         TIMESTAMPTZ NOT NULL,
    renewal_target_at TIMESTAMPTZ NOT NULL,   -- computed at issuance, jittered
    renewed_from       BYTEA,                  -- serial_number of the predecessor cert
    status             TEXT        NOT NULL DEFAULT 'active'
        CHECK (status IN ('active', 'renewed', 'revoked', 'expired'))
);

CREATE INDEX ON certificates (not_after) WHERE status = 'active';
CREATE INDEX ON certificates (renewal_target_at) WHERE status = 'active';
CREATE INDEX ON certificates (service_id, issued_at DESC);

-- Service identity registry: what identity is a given workload allowed to request
CREATE TABLE services (
    service_id        TEXT PRIMARY KEY,
    allowed_identity  TEXT NOT NULL UNIQUE,   -- the one SPIFFE ID/SAN this service may request
    owning_team       TEXT NOT NULL,
    registered_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    decommissioned_at TIMESTAMPTZ
);

-- Issuance audit log: append-only, one row per attempted issuance regardless of outcome
CREATE TABLE issuance_audit_log (
    id                BIGSERIAL   PRIMARY KEY,
    service_id        TEXT        NOT NULL,
    requested_identity TEXT       NOT NULL,
    challenge_type    TEXT        NOT NULL,
    outcome           TEXT        NOT NULL CHECK (outcome IN ('issued', 'denied', 'error')),
    denial_reason     TEXT,
    serial_number     BYTEA,
    occurred_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX ON issuance_audit_log (service_id, occurred_at DESC);
CREATE INDEX ON issuance_audit_log (outcome, occurred_at DESC) WHERE outcome != 'issued';

certificates is partitioned by not_after in monthly ranges, since nearly every hot query (find everything expiring soon, purge everything long expired) is a range scan on that column, and it keeps the active partition small even as historical certificates accumulate for audit purposes over years. issuance_audit_log is append-only and partitioned by occurred_at, since it exists for compliance and incident forensics, not for serving live traffic, so older partitions can move to cheaper storage without touching the hot path.

A certificate row is born the moment the ACME Issuance Service finalizes an order, with status='active' and a renewal_target_at already computed with jitter. It transitions to status='renewed' the moment its successor is issued, which the data flow diagram above traces end to end, rather than being deleted, since the audit trail needs the full lineage of a service’s certificate history. It transitions to status='revoked' if an operator or an automated compromise-detection system calls the revoke path, which also inserts a row into revoked_certificates. Anything left in status='active' past its not_after gets swept to status='expired' by a background job, which should almost never fire in practice, since renewal should always beat expiry.

Key Algorithms and Protocols

The ACME Attestation Challenge

Covered in the ACME Issuance Service section above, the attestation-01 challenge is the algorithmic core of issuance authorization: it maps a signed, time-bound workload identity assertion to a specific allowed certificate subject, and rejects anything that does not match exactly.

Renewal Scheduling with Jitter

Covered in the Expiry Monitoring section above, the renewal scheduler’s core algorithm is a jittered early-trigger with exponential backoff on failure, which runs in O(1) time per certificate and needs only the issuance and expiry timestamps as state.

mTLS Peer Identity Verification

Holding a certificate signed by a trusted CA proves a peer’s identity was once vouched for. It does not, by itself, prove that peer is authorized to talk to you. A payments service and a marketing-emails service might both hold perfectly valid certificates from the same internal CA; mTLS identity verification is the step that stops the marketing service from calling an endpoint it was never meant to reach, by checking the peer certificate’s subject against an explicit allowlist rather than just checking that the chain validates.

// mTLS peer identity verification, layered on top of standard chain validation
// Demonstrates: extracting a SPIFFE-style URI SAN and checking it against
// a per-endpoint allowlist, not just checking "is this cert valid"
package mtls

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
)

type IdentityPolicy struct {
	AllowedCallers map[string][]string // endpoint -> list of allowed SPIFFE IDs
}

func (p *IdentityPolicy) VerifyPeer(endpoint string, state tls.ConnectionState) error {
	if len(state.PeerCertificates) == 0 {
		return fmt.Errorf("mTLS required, no peer certificate presented")
	}
	leaf := state.PeerCertificates[0]

	peerIdentity, err := extractSPIFFEURI(leaf)
	if err != nil {
		return fmt.Errorf("peer cert missing SPIFFE URI SAN: %w", err)
	}

	allowed, ok := p.AllowedCallers[endpoint]
	if !ok {
		return fmt.Errorf("no identity policy configured for endpoint %s, denying by default", endpoint)
	}
	for _, id := range allowed {
		if id == peerIdentity {
			return nil // authorized
		}
	}
	return fmt.Errorf("caller identity %s not authorized for %s", peerIdentity, endpoint)
}

func extractSPIFFEURI(cert *x509.Certificate) (string, error) {
	for _, uri := range cert.URIs {
		if uri.Scheme == "spiffe" {
			return uri.String(), nil
		}
	}
	return "", fmt.Errorf("no spiffe:// URI SAN present")
}

Complexity here is dominated by chain validation, which Go’s standard library does in effectively O(depth of chain), negligible for a two-tier hierarchy, plus an O(1) map lookup for the identity policy check. The edge case worth naming: default-deny. An endpoint with no configured policy must reject every caller, not allow every caller, or a missing config entry silently turns into an open door.

Scaling and Performance

The ACME Issuance Service and the Intermediate CA signing path are the two components under the most sustained load, and both scale differently: the issuance service is stateless and scales horizontally behind a load balancer, while the signing operation itself is bottlenecked by the HSM or KMS’s signing throughput, which does not scale by adding more application pods.

Capacity Estimation:

Given:
  - 10,000 services, ~15,000 active leaf certificates (multi-SAN accounted for)
  - Certificate lifetime: 7 days, renewal target at ~60% elapsed with +/-10% jitter
  - Steady-state issuance target: 50/sec, burst target: 500/sec
  - Certificate + chain size: ~2.5 KB (leaf + intermediate, PEM)

Steady-state renewal rate:
  15,000 certs / 7 days = ~2,143 renewals/day = ~0.025/sec average
  Spread by jitter across a ~1.4 day window per cert, so real sustained
  load is well under the 50/sec provisioned target - the 50/sec ceiling
  exists for burst absorption, not steady state.

Burst scenario (new region bootstrap, 10,000 services in 20 minutes):
  10,000 certs / 1,200 seconds = ~8.3 issuances/sec required
  Provisioned burst capacity of 500/sec gives ~60x headroom for
  simultaneous bootstraps across multiple regions plus retry traffic.

HSM/KMS signing throughput (the real ceiling):
  Typical cloud KMS asymmetric signing: ~200-1,000 signs/sec per key,
  depending on algorithm (ECDSA faster than RSA-2048).
  At 500/sec burst target, provision at least 2 intermediate CA keys
  per region behind the issuance service to stay under per-key limits.

Certificate registry storage (1-year audit retention):
  15,000 active + ~2,143 renewals/day * 365 days = ~795,000 rows/year
  795,000 rows * ~400 bytes/row (metadata only, not the cert bytes) = ~320 MB/year

OCSP responder load:
  Assume 5% of live mTLS connections do an explicit OCSP check (most rely
  on stapling) at 50,000 handshakes/sec fleet-wide = 2,500 OCSP checks/sec
  Each check is a single indexed serial_number lookup, sub-millisecond,
  easily horizontally scaled behind a read-replica pool.

CRL size:
  Revocation is rare by design (short-lived certs absorb most compromise
  risk). Assume 0.5% of active certs are ever revoked before expiry = 75 certs
  75 * ~40 bytes/entry = 3 KB - trivially cached at the edge, refreshed every 15 min.

The dominant bottleneck is not database throughput or network bandwidth, it is the asymmetric signing operation itself, since every issuance and renewal requires a private-key operation on the intermediate CA’s key, and that key deliberately cannot be exported to scale out arbitrarily. Provisioning multiple intermediate CA keys per region, each behind the same issuance service, is the standard lever: it distributes signing load without duplicating the trust root, since all intermediates still chain up to the same offline root. Caching is concentrated on the read-heavy paths: OCSP stapling on the resource-server side, aggressive CDN-style caching on /jwks-equivalent trust bundle endpoints, and a hot in-memory cache of the service registry inside the issuance service, since it changes far less often than it is read.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Intermediate CA key compromise	Anomalous issuance volume alert, or manual incident report	Attacker can mint valid leaf certs for any identity until detected	Revoke the intermediate certificate, publish updated CRL/OCSP, stand up a new intermediate signed by the (still offline) root, force renewal fleet-wide against the new intermediate
ACME Issuance Service outage	Health check failures, renewal retry-exhaustion alerts from Cert Agents	New issuance and renewal blocked; existing certs keep working until their own expiry	Failover to a healthy region’s issuance service; Cert Agents retry with backoff, comfortably inside the multi-day renewal margin
Cert Agent crash or stuck process	Expiry Monitor’s backstop scan finds no recent renewal for that service’s cert	That one service’s certificate silently stops renewing	Expiry Monitor pages on-call once inside 24h of expiry with no successful renewal; on-call restarts the agent or forces a manual issuance
Clock skew between a service host and the CA	Certificate not_before validation failures spike in logs	Freshly issued certs rejected as not-yet-valid, or renewal computed against a wrong local clock	NTP-sync all hosts; issuance service backdates not_before by a small buffer (a few minutes) to absorb minor skew
OCSP responder overload during a mass-revocation event	Response latency alarm on the OCSP responder pool	Verifiers relying on synchronous OCSP checks see timeouts, some may fail open or fail closed depending on config	Horizontally scale the OCSP responder pool (stateless, reads from a replica); ensure clients are configured to fail closed for mTLS-critical paths
Renewal thundering herd from a mis-set jitter window	Sudden spike in issuance QPS correlated with a specific deploy batch	Issuance service latency degrades fleet-wide right as the spike hits	Rate-limit per-service issuance requests; alert if jitter variance collapses (a sign the jitter computation itself has a bug)

Comparison of Approaches

Approach	Latency per issuance	Complexity	Failure mode	Best fit
Internal CA + ACME automation (this design)	Sub-second, fully automated	High (hierarchy, HSM, attestation, scheduler)	Intermediate compromise requires coordinated fleet-wide re-issuance	Large internal service fleets (1,000+ services) needing zero-touch rotation
Long-lived certs, manual issuance	Minutes to days (human in the loop)	Low (no automation infra needed)	Expiry outages from forgotten renewals; large blast radius on key leak	Small fleets (tens of services) with infrequent changes
Public CA (Let’s Encrypt) for internal services	Seconds, automated via public ACME	Low-medium (no CA infra to run)	Requires internet-reachable validation (HTTP-01/DNS-01), awkward for internal-only hostnames	Internet-facing endpoints, or hybrid setups where internal services also need public trust
Cloud-managed PKI (e.g. a managed private CA service)	Sub-second, automated, provider-managed HSM	Medium (less infra to run, but vendor lock-in and per-cert cost)	Provider outage blocks issuance fleet-wide; less control over challenge customization	Teams that want the automation benefits without operating HSMs themselves
Self-signed certs per service, manually distributed trust	Instant to generate, slow to distribute trust	Low to build, high to operate at scale	No central revocation; distributing updated trust bundles to every verifier is fully manual	Prototypes, single-team internal tools, not production fleets

We would pick the internal CA plus ACME automation approach for any fleet past a few hundred services, because the alternative approaches all fail on one of the two axes that matter at this scale: either issuance requires a human (long-lived manual certs, self-signed certs) or trust is anchored somewhere we do not control and cannot customize the identity-validation challenge for (public CA, most managed PKI offerings). The upfront cost of building the hierarchy, the attestation challenge, and the renewal scheduler pays for itself the first time a certificate would have silently expired at 3am under a manual process and instead just quietly renewed on schedule.

Key Takeaways

• The two-tier CA hierarchy separates the most valuable key from the busiest key. The offline root signs only intermediates and stays cold; the online intermediate does the daily signing work and can be rotated without touching the root.
• Short certificate lifetimes (7 days) shrink the blast radius of a key leak and shift the system’s safety margin away from perfect revocation and toward continuous, automated renewal.
• The attestation-01 ACME challenge binds signature validity, freshness, and an exact identity-registry match together, and weakening any one of the three breaks the whole authorization guarantee.
• Hot reload via an atomic in-memory pointer swap, not a process restart or signal handler, is what makes zero-downtime rotation actually zero-downtime.
• Jitter computed once at issuance time, not recomputed on every scheduler tick, is what keeps 10,000 services from renewing in lockstep.
• mTLS chain validation proves identity, not authorization. A per-endpoint identity allowlist is a separate, necessary check on top of “the certificate is valid.”
• The signing operation on the intermediate CA’s key, not the database or the network, is the real scaling ceiling, and the lever is provisioning more intermediate keys, not more application pods.
• Revocation infrastructure matters more, not less, because certificates are short-lived, since the rare cases that do need revocation (a compromised intermediate) are exactly the highest-severity ones.

The counter-intuitive lesson is that making certificates expire faster is what makes the whole system safer, not riskier, provided renewal is fully automated. A team’s first instinct is often to lengthen certificate lifetimes to reduce operational load, when the actual fix for operational load is automating renewal so thoroughly that lifetime length stops being an operational concern at all, freeing you to shorten it purely for security reasons.

Frequently Asked Questions

Q: Why not just issue certificates with a one-year lifetime like most public CAs use, to reduce how often anything has to renew?

A: A one-year certificate turns a single leaked private key into a year-long exposure window, and it means your renewal automation, if it has a latent bug, might not surface that bug for months. Short-lived certificates force renewal automation to be exercised constantly, so bugs in it show up in days, not months, and they bound how long any single compromised key stays useful. The tradeoff is that renewal absolutely must be automated and reliable, which is exactly the system this post designs.

Q: Why not skip the internal CA entirely and just get every service a certificate from Let’s Encrypt or another public CA?

A: Public ACME validation methods (HTTP-01, DNS-01) assume the identifier being proven is a domain name reachable from the internet or a DNS zone you can modify. Most internal service identities (a Kubernetes service account, a SPIFFE ID, an internal-only hostname) do not map cleanly onto that model, and routing internal validation traffic out to the public internet just to prove internal identity is both awkward and an unnecessary external dependency for something that should work even if the internet connection is down.

Q: Why maintain both OCSP and a CRL instead of picking one revocation mechanism?

A: They serve different verifier shapes. OCSP is a per-certificate, low-latency lookup, ideal for a live handshake path, especially with stapling so the client never makes the call itself. A CRL is a bulk, cacheable list, better for verifiers that check many certificates locally without wanting a network round trip per check, or that need to keep working during an OCSP responder outage. Running both costs relatively little once the underlying revocation table exists, since both are just different serializations of the same data.

Q: Why generate the private key on the service host instead of centrally, where it would be easier to manage and back up?

A: A centrally generated key has to be transported to the host that will use it, and every transport step is a chance for the key to be logged, cached, or intercepted somewhere it shouldn’t be. Generating the key locally means the private key material never exists anywhere except the process memory of the host that needs it; only the public key (embedded in the CSR) ever travels over the network. The cost is that key backup and recovery have to be designed per-host rather than centrally, which is an acceptable tradeoff given that a lost key just triggers a new issuance, not data loss.

Q: What stops a compromised Cert Agent from requesting a certificate for an identity it shouldn’t have?

A: The attestation challenge, not the ACME protocol’s default trust model. The CSR’s requested identity is cross-checked against the service registry entry tied to a freshly verified attestation document proving which specific workload is asking. A compromised host can still request a certificate for its own legitimate identity (which is a real risk, but bounded to that one identity), but it cannot successfully request a certificate for a different service’s identity, because the attestation would not match the registry’s expected mapping for that other identity.

Q: How would you handle a service that legitimately needs a much shorter certificate lifetime than 7 days, like a highly sensitive payments path?

A: Certificate lifetime is a per-identity policy in the service registry, not a global constant. A sensitive service can be configured with a shorter lifetime (a day, or even hours) and a correspondingly earlier renewal target; the ACME Issuance Service reads that policy when finalizing the order rather than applying one fixed duration fleet-wide. The tradeoff is more issuance traffic for that service specifically, which is usually an acceptable cost for the services where it actually matters.

Interview Questions

Q: Walk through what happens end to end when the intermediate CA’s signing key is suspected compromised, from detection to the fleet being safe again.

Expected depth: Cover detection (anomalous issuance volume, unexpected identities, or an external report), immediate containment (revoke the intermediate certificate itself, which invalidates trust in everything it signed going forward for new verifications), publishing that revocation via both OCSP and an updated CRL, standing up a replacement intermediate signed by the still-offline root (a deliberate, rare operation), and then forcing every one of 10,000 services to re-issue against the new intermediate before their current (now-suspect) certificates would have naturally expired. Discuss why short certificate lifetimes reduce, but do not eliminate, the urgency here.

Q: Design the renewal scheduler so that 10,000 services never cause a thundering herd against the issuance service, even if they were all provisioned in the same deployment.

Expected depth: Explain jitter computed once at issuance time and stored (not recomputed per scheduler tick), the tradeoff between renewal threshold (how early) and jitter window (how spread out), and how to size the issuance service’s steady-state capacity against the actual sustained renewal rate versus its burst capacity. Discuss what happens if jitter is implemented incorrectly (e.g., reseeded per process restart in a way that correlates across hosts restarted together) and how you’d detect that failure mode in production metrics.

Q: How would you extend this system to support certificate issuance for services running outside your own infrastructure, like a third-party vendor integration that also needs mTLS?

Expected depth: Discuss why the attestation-01 challenge (which assumes a trusted internal attestor like a cloud provider or SPIRE agent) does not extend cleanly to an external party, and what alternative challenge or bootstrap mechanism would be needed (a manually provisioned, tightly scoped bootstrap credential, or a separate lower-trust intermediate CA specifically for external identities so a compromise there cannot be used to impersonate internal services).

Q: A service team asks why their mTLS connection is being rejected even though openssl verify confirms their peer’s certificate chains correctly to the trusted root. What do you check?

Expected depth: Distinguish chain validation (which openssl verify checks) from authorization (the per-endpoint identity allowlist described in the mTLS Peer Identity Verification section). Walk through checking whether the peer’s SPIFFE URI SAN or subject matches what the destination endpoint’s policy actually allows, whether the policy exists at all for that endpoint (default-deny misconfiguration), and whether OCSP stapling is returning a stale or missing status causing a fail-closed rejection.

Q: How would you redesign the OCSP responder if you needed it to survive a full regional outage of your primary database without serving stale-but-wrong “good” responses for certificates revoked in another region during the outage?

Expected depth: Discuss replicating the revocation table (a small, append-mostly dataset) to every region rather than centralizing OCSP behind one region, the tradeoff of eventual consistency in that replication versus the propagation SLA, and the design choice of a conservative default (a responder that cannot confirm a certificate’s status recently enough might return “unknown” rather than “good,” pushing the decision to the verifier’s fail-open/fail-closed policy) rather than silently trusting potentially stale local state.