Think of a bank vault with safety deposit boxes. The bank holds a master key that opens the vault door. Each customer holds a key that opens only their box. A bank employee cannot read your box without your key even if they have unrestricted physical access to the vault. A thief who steals the vault itself cannot open any box without both the master key and each individual customer key. This layered protection is the core intuition behind secrets management at scale.

At the scale of a real production system, hundreds of microservices each need dozens of credentials: database passwords, API keys for third-party providers, signing keys for JWTs, TLS private keys, OAuth client secrets. Credentials are accessed thousands of times per second. Some are long-lived static secrets that need rotating on a schedule. Others are ephemeral dynamic secrets generated on demand and valid for one hour. A single leaked credential in a shared environment has a blast radius proportional to how many services use it.

Three tensions dominate the design. Security demands encrypt-everything and audit-everything, producing an immutable record of every access. Performance demands sub-5ms read latency on the hot path because credentials are checked at service startup and on every authenticated API call. Availability demands that if the vault is down, every service that reads credentials at startup fails to start, which turns a vault outage into a full platform outage.

We need to solve for a key hierarchy that limits blast radius so that compromising one layer does not expose all secrets, a policy engine that enforces least-privilege without becoming a bottleneck at 10,000 requests per second, and lease-based rotation that updates consumers before revoking the old credential with no service restart required.

Requirements and Constraints

Functional Requirements:

• CRUD secrets at arbitrary path hierarchies (secret/prod/db/postgres, secret/staging/api/stripe)
• Dynamic secrets: generate database credentials on demand with a TTL; no pre-provisioned shared passwords
• Static secret auto-rotation: trigger rotation before TTL expiry, notify consumers via watch endpoint
• Full audit trail: every read, write, delete, and list operation is logged with token identity and timestamp
• RBAC: tokens bound to policies; policies express path patterns and capability ACLs
• Seal/unseal: vault starts sealed with the master key encrypted; K-of-N key shards required to unseal

Non-Functional Requirements:

• p99 read latency under 5ms
• 10,000 req/s sustained throughput
• 99.99% availability (vault down means services cannot start)
• AES-256-GCM encryption at rest
• TLS 1.3 in transit, mTLS between internal components
• Audit logs retained 1 year minimum
• RPO under 1 second, RTO under 30 seconds

Constraints:

• API-first; not a secrets browser for human operators
• Secret payload capped at 512 KB
• No cross-region synchronous writes (async DR replication only)

High-Level Architecture

Seven components make up the system. The API Gateway handles TLS termination, rate limiting, and Bearer token extraction. The Auth and Policy Engine validates token identity and evaluates RBAC policies per request. The Vault Core orchestrates business logic: envelope encryption, decryption, and secret versioning. The Lease Manager tracks TTLs, triggers rotation for static secrets, and drops dynamic credentials when their lease expires. The Storage Backend persists encrypted blobs in a Raft-replicated store - it never sees plaintext. The KMS/HSM holds the root key material and performs key wrapping and unwrapping operations. The Audit Pipeline receives a write-ahead event before every response is returned, writing to an immutable append-only log.

Data flows like this for a read request: the client sends a Bearer token and a secret path to the API Gateway. The Auth Engine validates the token identity, evaluates the token’s bound policies against the requested path and operation, and passes an authenticated request to Vault Core. Vault Core fetches the encrypted blob and its associated key identifier from Storage. It sends the encrypted DEK to KMS for unwrapping. KMS returns the plaintext DEK without ever seeing the secret payload. Vault Core decrypts the blob in memory using the DEK and returns the plaintext to the client. Before returning, it writes an audit event to the Audit Pipeline.

The Lease Manager runs a background sweep every second. For static secrets, it checks whether any secret’s TTL is within the rotation window and triggers rotation if so. For dynamic secrets like ephemeral database users, it calls into the backend provider (Postgres, MySQL, AWS IAM) to drop the credential when the lease expires. Every successful access also records a lease so that explicitly revoked tokens immediately cut off access even if the in-process cache would otherwise have served the old value.

The Encryption Layer