Picture a hotel’s electronic keycard system. The front desk programs a card for a specific guest with a specific set of doors it will open: room 412, the gym, the pool, never the penthouse and never the housekeeping closets. If that guest loses the card in the lobby, the front desk cannot wait until checkout to deal with it. The lost card has to stop opening every single door in the building within seconds, not just the door the front desk happens to be standing next to, and every other guest’s card has to keep working exactly as before. An API key management system is that hotel’s access-control backend rebuilt for software: it issues credentials scoped to exactly what a client should be allowed to do, and when one leaks or gets revoked, that decision has to reach every enforcement point almost instantly.

Consider a platform issuing keys to roughly 5 million active third-party integrations, fielding 60,000 requests a second on average and spiking past 200,000 requests a second during a partner’s own peak traffic, spread across a gateway fleet of 150-plus edge nodes in multiple regions. Every one of those requests carries an API key in a header, and every one has to be authenticated, checked against the scopes the key was actually granted, and counted against that key’s own rate limit, all before the request is allowed to touch the origin service. When a key is compromised or a customer rotates off a leaked credential, the platform has committed to shutting that key down everywhere, on every one of those 150 gateway nodes, within one second.

The naive approach is a synchronous database lookup on every request: hash the incoming key, SELECT the row, check status = 'active', check the scopes column, and let the request through. That fails the moment you do the math. At 200,000 requests a second, a single authoritative datastore taking even a single row lookup per request becomes the busiest system in the entire architecture, and it sits directly in the latency budget of every request the platform serves, including the ones that have nothing to do with key management. Push authorization checks onto a synchronous, centralized dependency and you have turned an internal bookkeeping system into the platform’s most fragile single point of failure.

The second failure mode shows up once you try to fix the first one with a plain cache. Cache the key’s status at the gateway with a generous TTL, say five minutes, and the hot path gets fast again, but now a revoked key keeps working at any gateway node holding a stale cache entry for up to five minutes after the platform believes it has been shut down. Shrink the TTL to make revocation faster and you are back to hammering the database, just on a tighter cycle. Neither a synchronous check nor a long-TTL cache alone satisfies both constraints at once. We need to solve for three things simultaneously: authorizing a scoped request off a cache that lives entirely on the gateway’s own hot path, propagating a revocation to every gateway node inside a hard one-second budget without a synchronous read on every request, and recording detailed per-key usage for billing and analytics without that recording ever being on the critical path of the response.

Requirements and Constraints

Functional Requirements

• Issue API keys scoped to specific resources and actions (orders:read, orders:write, webhooks:manage), with a key never usable outside its granted scopes
• Verify a key and its scopes on every incoming request and reject anything outside its granted scope set with a 403
• Enforce a distinct rate limit per key (requests per second and a burst allowance), configurable per plan tier, independent of any other key’s traffic
• Revoke a key on demand (manual action, automated leak detection, customer offboarding) and guarantee that revocation is honored by every gateway node within 1 second
• Support key rotation with a grace period so a client can adopt a new secret before the old one stops working, with zero downtime for the integration
• Record usage per key (request count, endpoint, status code, bytes transferred) for billing and for a per-key usage dashboard
• Expose an admin API to create, list, scope, rotate, and revoke keys, and a read API for customers to see their own key’s usage

Non-Functional Requirements

• Hot path latency: authorization and rate-limit check add no more than 2ms at p99 to a request’s total latency, since this sits in front of every API call the platform serves
• Throughput: sustain 200,000 requests/second at peak across the gateway fleet, 60,000/second sustained average, each individually authenticated and rate-limited
• Scale: 5 million active keys, roughly 150 gateway nodes across 3+ regions, around 300 million usage events per day
• Revocation SLA: a revoked key must stop being accepted on every gateway node within 1 second of the revocation being issued, with no manual intervention
• Availability: 99.99% for the authorization path; the gateway must fail toward a safe, bounded default (serve from last-known-good cache, never block all traffic) if the control plane is unreachable
• Durability: key records and revocation history are never lost; usage analytics can tolerate seconds of delay but not silent data loss
• Consistency: strong consistency for the authoritative key record in the control plane, bounded eventual consistency (sub-1s) for gateway-side enforcement

Constraints and Assumptions

• We assume TLS terminates before the gateway and focus on application-layer key validation, not transport security
• We do not design the customer-facing dashboard UI in depth, only the APIs and data model behind key management and usage reporting
• OAuth2 token flows (short-lived bearer tokens, refresh tokens) are out of scope; this is a long-lived, directly-presented API key model, the pattern used by Stripe, GitHub, and most B2B platform APIs
• A single global control plane with regionally replicated read paths is assumed; we discuss cross-region propagation only where the revocation SLA depends on it
• We do not enumerate every possible scope taxonomy for every API surface; the goal is the mechanism for scoped authorization, not a fixed list of scopes for any one product

High-Level Architecture

The system has six major components: the Key Issuance Service, the Gateway Auth Middleware (with its local cache), the Per-Key Rate Limiter, the Revocation Propagation layer, the Usage Event Pipeline, and the Key Rotation Workflow.

The Key Issuance Service is the control plane: it generates keys, hashes them for storage, assigns scopes and rate-limit tiers, and is the single source of truth for a key’s current status. The Gateway Auth Middleware runs on every edge node in front of the origin services; it extracts a key from the request, checks it against a local, short-TTL cache before ever considering a call back to the control plane, and enforces scopes. The Per-Key Rate Limiter is a sharded counter store, typically Redis, that the gateway calls after a key passes authentication, tracking request counts per key independently of every other key. Revocation Propagation is the mechanism that pushes a “this key is no longer valid” event out to all 150-plus gateway nodes the instant an admin or an automated detector revokes a key. The Usage Event Pipeline receives a lightweight, fire-and-forget event from the gateway after every request and rolls it up asynchronously into per-key usage analytics. The Key Rotation Workflow lets a key holder generate a new secret while the old one keeps working for a bounded overlap window, so rotating a credential never requires a synchronized cutover.

A request flows like this: a client calls the API with a key in the Authorization header. The gateway hashes the key, checks its local in-memory cache for that key’s metadata (status, scopes, rate-limit config), and on a cache hit, which is the overwhelming majority of requests, never talks to any other service to authenticate the request. It checks the requested scope against the key’s granted scopes, calls the rate limiter with the key’s identifier, and if both checks pass, forwards the request to the origin and fires an async usage event. On a cache miss, the gateway fetches the key’s metadata from the control plane once, populates its local cache, and proceeds. Separately, whenever a key’s status changes, the Key Issuance Service publishes a revocation event that every gateway node subscribes to, invalidating that key in every local cache within the propagation SLA regardless of each node’s own cache TTL.

Component Deep Dives

The Key Issuance Service

This component’s job is to be the single authoritative source for what a key is allowed to do and whether it is still valid, and to never hand out or store a key’s raw secret after the moment it is created.

Key hashing for storage means the platform stores a one-way hash of the secret, never the secret itself, so that a database breach does not hand an attacker a list of usable credentials. A generated key looks like sk_live_7hQ3mZ..., a static prefix plus a long random secret; the service stores the prefix (for display and lookup) and an HMAC-SHA256 hash of the full secret, keyed with a server-side pepper that never lives in the database.

# Key generation and hashing at issuance time
# Demonstrates: cryptographically random secret, HMAC hash with a server-side pepper,
# constant-time verification so timing differences never leak information about the hash
import hashlib
import hmac
import secrets

PEPPER = b"loaded-from-a-secrets-manager-not-the-database"  # never stored alongside key_hash

def generate_api_key(env: str = "live") -> tuple[str, str, bytes]:
    raw_secret = secrets.token_urlsafe(32)          # ~256 bits of entropy
    prefix = f"sk_{env}_{raw_secret[:8]}"
    full_key = f"sk_{env}_{raw_secret}"
    key_hash = hmac.new(PEPPER, full_key.encode(), hashlib.sha256).digest()
    return full_key, prefix, key_hash               # full_key is shown to the caller exactly once

def verify_api_key(candidate_key: str, stored_hash: bytes) -> bool:
    candidate_hash = hmac.new(PEPPER, candidate_key.encode(), hashlib.sha256).digest()
    # compare_digest runs in constant time regardless of where the strings first differ,
    # which matters because a timing side channel would let an attacker guess a hash byte by byte
    return hmac.compare_digest(candidate_hash, stored_hash)

Scope-based authorization is assigned at issuance and stored as a compact set on the key record: a scope like orders:write names a resource and an action, and a key only ever gets the scopes its owner explicitly grants it. The prefix is deliberately not secret; it is what shows up in a “last used” dashboard and what a leak-detection scanner matches against, while the secret half never round-trips back to the server in cleartext after creation.

Think of the prefix and secret split like a hotel room number printed on the outside of a keycard sleeve next to the card itself: the room number is not sensitive, it just tells the desk clerk which card this is at a glance, while the actual magnetic strip is what a locked door checks. If you simplified this component to storing keys in plaintext, or to a reversible encryption scheme instead of a one-way hash, a single database dump would hand an attacker every live credential on the platform at once, which is exactly the class of breach key_hash with HMAC and a pepper is designed to survive.

The Gateway Auth Middleware and Local Cache

This component’s job is to answer “is this key valid, and for this scope” on every single request, in well under a millisecond, without becoming a bottleneck itself.

Local cache with TTL is what makes that possible. Each gateway node keeps an in-process cache mapping a key’s hash to its metadata (status, scopes, rate-limit tier, revocation epoch), with a TTL long enough to avoid hammering the control plane, typically 30 to 60 seconds, and short enough that a cache entry naturally self-heals if it somehow misses a revocation push. The TTL is a backstop, not the primary revocation mechanism.

// Gateway-side local key cache with TTL and epoch-aware invalidation
// Demonstrates: sub-millisecond lookup, background refresh on miss, revocation epoch check
package authcache

import (
	"sync"
	"time"
)

type KeyMeta struct {
	Status          string   // "active" or "revoked"
	Scopes          []string
	RateLimitRPS    int
	RateLimitBurst  int
	RevocationEpoch int64
	CachedAt        time.Time
}

type LocalCache struct {
	mu    sync.RWMutex
	byKey map[string]KeyMeta
	ttl   time.Duration
}

func NewLocalCache(ttl time.Duration) *LocalCache {
	return &LocalCache{byKey: make(map[string]KeyMeta), ttl: ttl}
}

// Get returns (meta, true) only if present and not past its TTL; callers fall
// back to a control-plane fetch on a miss, which also refreshes the entry.
func (c *LocalCache) Get(keyHash string) (KeyMeta, bool) {
	c.mu.RLock()
	defer c.mu.RUnlock()
	meta, ok := c.byKey[keyHash]
	if !ok || time.Since(meta.CachedAt) > c.ttl {
		return KeyMeta{}, false
	}
	return meta, true
}

func (c *LocalCache) Set(keyHash string, meta KeyMeta) {
	meta.CachedAt = time.Now()
	c.mu.Lock()
	defer c.mu.Unlock()
	c.byKey[keyHash] = meta
}

// Invalidate is called the instant a revocation event arrives over the pub/sub
// subscription, regardless of how much of the TTL window remains.
func (c *LocalCache) Invalidate(keyHash string, epoch int64) {
	c.mu.Lock()
	defer c.mu.Unlock()
	if meta, ok := c.byKey[keyHash]; ok && epoch >= meta.RevocationEpoch {
		meta.Status = "revoked"
		meta.RevocationEpoch = epoch
		c.byKey[keyHash] = meta
	}
}

The scope check itself is a simple set membership test against the cached scopes, cheap enough to run on every request without measurable overhead. The mistake a smart engineer often makes here is assuming the cache TTL is the revocation mechanism; treated that way, a 60-second TTL means a revoked key can still be accepted anywhere from 0 to 60 seconds after revocation, which blows well past a 1-second SLA. The TTL exists purely so a gateway node that somehow missed a push event (a restart, a brief network partition to the pub/sub broker) does not serve stale data forever; the push path, covered next, is what actually meets the SLA.

The Per-Key Rate Limiter

This component’s job is to enforce that one key’s traffic never borrows capacity from, or gets throttled by, any other key’s traffic, at 200,000 requests a second in aggregate.

Per-key rate limit counters use a token bucket per key: each key has a bucket that refills at its configured rate (say 50 requests/second) up to a burst ceiling (say 100), and every request consumes one token if available or gets rejected with a 429 if the bucket is empty. A token bucket, unlike a fixed window counter, tolerates a legitimate short burst without either starving a bursty client or letting a sustained abuser through, because burst capacity and sustained rate are configured independently.

-- Token bucket rate limiter, atomic via a single Redis Lua script
-- Demonstrates: O(1) check-and-consume, lazy refill (no background timer needed)
-- KEYS[1] = bucket key, ARGV[1] = rate_per_sec, ARGV[2] = burst, ARGV[3] = now_ms
local bucket_key = KEYS[1]
local rate = tonumber(ARGV[1])
local burst = tonumber(ARGV[2])
local now_ms = tonumber(ARGV[3])

local data = redis.call('HMGET', bucket_key, 'tokens', 'ts')
local tokens = tonumber(data[1])
local last_ts = tonumber(data[2])

if tokens == nil then
  tokens = burst
  last_ts = now_ms
end

-- Lazily refill based on elapsed time since the last request, rather than
-- running a background job per key to top up every bucket on a timer
local elapsed_sec = math.max(now_ms - last_ts, 0) / 1000.0
tokens = math.min(burst, tokens + elapsed_sec * rate)

local allowed = 0
if tokens >= 1.0 then
  tokens = tokens - 1.0
  allowed = 1
end

redis.call('HMSET', bucket_key, 'tokens', tokens, 'ts', now_ms)
redis.call('EXPIRE', bucket_key, math.ceil(burst / rate) + 5)
return allowed

// Go wrapper calling the token bucket script, keyed per API key
// Demonstrates: sharding rate-limit keys across a Redis cluster by key hash
package ratelimit

import (
	"context"
	"strconv"
	"time"

	"github.com/redis/go-redis/v9"
)

func Allow(ctx context.Context, rdb redis.Scripter, script *redis.Script, keyID string, ratePerSec, burst int) (bool, error) {
	bucketKey := "rl:" + keyID
	nowMs := strconv.FormatInt(time.Now().UnixMilli(), 10)
	res, err := script.Run(ctx, rdb, []string{bucketKey}, ratePerSec, burst, nowMs).Int()
	if err != nil {
		return false, err
	}
	return res == 1, nil
}

Bucket keys are sharded across the Redis cluster by hashing the key’s identifier, exactly the way the Reputation Store shards by IP in other rate-limiting designs, so no single Redis node ever has to hold or serve every key’s bucket. If you simplified this to one global counter per gateway node instead of a shared distributed bucket, a client with multiple concurrent connections spread across gateway nodes would get a multiple of its intended limit, since each node would be counting independently with no shared state.

Revocation Propagation

This component’s job is to make a revocation decision, made once at the control plane, visible and enforced on every gateway node within 1 second, without any gateway node polling the control plane on a tight loop.

Revocation propagation is built on a publish-subscribe fanout: the Key Issuance Service publishes a small message (key_hash, new status, a monotonically increasing revocation_epoch) to a broker that every gateway node subscribes to, and each node applies the invalidation to its local cache the instant the message arrives, independent of that entry’s remaining TTL.

# Control-plane side: revoke a key and publish the invalidation event
# Demonstrates: monotonic epoch (not a boolean flag) so out-of-order delivery
# across a pub/sub fanout can never revive a key that should stay revoked
import json
import time

def revoke_key(db_conn, redis_pubsub, key_hash: str, reason: str) -> int:
    epoch = int(time.time() * 1000)  # epoch doubles as a delivery-order tiebreaker
    db_conn.execute(
        "UPDATE api_keys SET status = 'revoked', revocation_epoch = %s, revoked_at = NOW() "
        "WHERE key_hash = %s",
        (epoch, key_hash),
    )
    db_conn.execute(
        "INSERT INTO revocation_log (key_hash, epoch, reason) VALUES (%s, %s, %s)",
        (key_hash, epoch, reason),
    )
    event = json.dumps({"key_hash": key_hash, "status": "revoked", "epoch": epoch})
    redis_pubsub.publish("key-revocations", event)  # fanout to every subscribed gateway node
    return epoch

# Gateway side: subscriber applies invalidations as they arrive, plus a
# periodic anti-entropy pull as a backstop against a missed pub/sub message
import json
import time

def revocation_subscriber_loop(pubsub, local_cache):
    for message in pubsub.listen():
        if message["type"] != "message":
            continue
        event = json.loads(message["data"])
        local_cache.invalidate(event["key_hash"], event["epoch"])

def anti_entropy_reconcile(db_conn, local_cache, last_checkpoint_epoch: int) -> int:
    # Runs every few seconds regardless of pub/sub health, catching any
    # revocation a node missed due to a reconnect or a dropped message
    rows = db_conn.execute(
        "SELECT key_hash, revocation_epoch FROM revocation_log WHERE epoch > %s ORDER BY epoch",
        (last_checkpoint_epoch,),
    ).fetchall()
    max_epoch = last_checkpoint_epoch
    for key_hash, epoch in rows:
        local_cache.invalidate(key_hash, epoch)
        max_epoch = max(max_epoch, epoch)
    return max_epoch

Using a monotonically increasing epoch instead of a plain boolean matters because pub/sub fanout offers no ordering guarantee across a fleet of 150 subscribers; a node that briefly reconnects could in principle receive a later message before an earlier one, and comparing epochs rather than blindly overwriting status is what prevents a stale, out-of-order message from reviving a key that should stay revoked. The anti-entropy pull is a deliberately cheap, low-frequency safety net, not the primary path, so it never becomes the bottleneck the push mechanism was built to avoid.

The Usage Event Pipeline

This component’s job is to capture what happened on every request for billing and analytics, without that capture ever being on the request’s own critical path.

Usage event pipeline means the gateway does not write usage data synchronously; it emits a small, fire-and-forget event after the decision to serve or reject a request has already been made, and a separate asynchronous system does the aggregation.

# Gateway-side usage event emission, fire-and-forget after the request completes
# Demonstrates: non-blocking emit, bounded local buffer, batched async flush to Kafka
import json
import time
from queue import Queue, Full

_buffer: Queue = Queue(maxsize=50_000)

def emit_usage_event(key_id: str, endpoint: str, status_code: int, bytes_out: int) -> None:
    event = {
        "key_id": key_id,
        "endpoint": endpoint,
        "status_code": status_code,
        "bytes_out": bytes_out,
        "ts": time.time(),
    }
    try:
        _buffer.put_nowait(event)   # never blocks the request thread
    except Full:
        pass                        # drop under extreme load rather than add latency

def flush_batch(producer, topic: str = "api-usage-events", max_batch: int = 500) -> int:
    batch = []
    while not _buffer.empty() and len(batch) < max_batch:
        batch.append(_buffer.get_nowait())
    if batch:
        producer.produce(topic, value=json.dumps(batch).encode())
        producer.flush(timeout=1.0)
    return len(batch)

A stream aggregator consumes the Kafka topic and rolls raw events up into per-key, per-minute counters, which is what the usage dashboard and the billing system actually query against, since nobody needs to scan 300 million raw events to answer “how many requests did key X make today”. If a spike drops some events under Full, that is an accepted tradeoff: a slightly under-counted analytics number is vastly preferable to adding synchronous write latency to every request just to guarantee analytics precision.

Key Rotation Workflow

This component’s job is to let a key holder replace a secret without a synchronized cutover, since real integrations cannot update every server and background job at the exact same instant.

Key rotation workflow issues a brand-new secret under the same logical key identity while the old secret keeps working for a bounded grace window, typically 7 to 30 days, after which the old secret is automatically revoked.

-- Rotation modeled as a linked pair of key rows sharing an owner, not an in-place secret swap
UPDATE api_keys
SET status = 'rotating', rotation_grace_expires_at = NOW() + INTERVAL '14 days'
WHERE key_hash = $1;

INSERT INTO api_keys (key_id, key_hash, key_prefix, owner_id, scopes, rate_limit_rps,
                       rate_limit_burst, status, rotated_from, created_at)
VALUES ($2, $3, $4, (SELECT owner_id FROM api_keys WHERE key_hash = $1),
        (SELECT scopes FROM api_keys WHERE key_hash = $1),
        (SELECT rate_limit_rps FROM api_keys WHERE key_hash = $1),
        (SELECT rate_limit_burst FROM api_keys WHERE key_hash = $1),
        'active', $1, NOW());

# Scheduled job: sweep keys whose rotation grace period has expired
# Demonstrates: automatic finalization of a rotation without a manual step
def expire_rotation_grace_periods(db_conn, revoke_fn) -> int:
    rows = db_conn.execute(
        "SELECT key_hash FROM api_keys "
        "WHERE status = 'rotating' AND rotation_grace_expires_at < NOW()"
    ).fetchall()
    for (key_hash,) in rows:
        revoke_fn(db_conn, key_hash, reason="rotation_grace_expired")
    return len(rows)

Both the old and new secret validate successfully during the grace window, each against its own row, both carrying identical scopes and rate limits inherited from the original key, so from the gateway’s point of view rotation looks like two ordinary active keys rather than a special case. Skip the grace window and force an instant swap, and every client using the old secret starts failing the moment the new one is issued, which turns a routine credential hygiene practice into an outage for anyone who has not redeployed yet.

Data Model

The data model spans four entities: API keys, the revocation log, usage rollups, and rate-limit configuration attached to each key.

-- API keys: the authoritative record; the raw secret is never stored, only its hash
CREATE TABLE api_keys (
    key_id                     UUID        PRIMARY KEY DEFAULT gen_random_uuid(),
    key_prefix                 TEXT        NOT NULL UNIQUE,   -- e.g. sk_live_7hQ3mZ1a, shown in UI
    key_hash                   BYTEA       NOT NULL UNIQUE,   -- HMAC-SHA256(secret, pepper)
    owner_id                   UUID        NOT NULL REFERENCES accounts(account_id),
    name                       TEXT        NOT NULL DEFAULT 'Untitled key',
    scopes                     TEXT[]      NOT NULL DEFAULT '{}',
    rate_limit_rps             INTEGER     NOT NULL DEFAULT 50,
    rate_limit_burst           INTEGER     NOT NULL DEFAULT 100,
    status                     TEXT        NOT NULL CHECK (status IN ('active','rotating','revoked')) DEFAULT 'active',
    revocation_epoch           BIGINT      NOT NULL DEFAULT 0,
    rotated_from                UUID        REFERENCES api_keys(key_id),
    rotation_grace_expires_at  TIMESTAMPTZ,
    expires_at                 TIMESTAMPTZ,
    created_at                 TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    revoked_at                 TIMESTAMPTZ
);

CREATE INDEX ON api_keys (owner_id);
CREATE INDEX ON api_keys (status) WHERE status != 'revoked';

-- Revocation log: append-only, the source of truth for gateway anti-entropy catch-up
CREATE TABLE revocation_log (
    id         BIGSERIAL   PRIMARY KEY,
    key_hash   BYTEA       NOT NULL,
    epoch      BIGINT      NOT NULL,
    reason     TEXT        NOT NULL CHECK (reason IN ('manual','leak_detected','rotation_grace_expired','owner_offboarded')),
    actor_id   UUID,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX ON revocation_log (epoch);

-- Usage rollups: pre-aggregated per key per minute, partitioned by day for cheap retention
CREATE TABLE usage_rollups_minute (
    key_id         UUID        NOT NULL,
    minute_bucket  TIMESTAMPTZ NOT NULL,
    request_count  BIGINT      NOT NULL DEFAULT 0,
    error_count    BIGINT      NOT NULL DEFAULT 0,
    bytes_out      BIGINT      NOT NULL DEFAULT 0,
    PRIMARY KEY (key_id, minute_bucket)
) PARTITION BY RANGE (minute_bucket);

CREATE TABLE usage_rollups_minute_2026_07_04 PARTITION OF usage_rollups_minute
    FOR VALUES FROM ('2026-07-04') TO ('2026-07-05');

CREATE INDEX ON usage_rollups_minute (minute_bucket);

api_keys is looked up almost exclusively by key_hash on the (rare) cache-miss path and by owner_id on the admin dashboard, so both get dedicated indexes while the table itself stays small relative to request volume, since it is one row per key, not per request. revocation_log is deliberately append-only and indexed on epoch, the exact shape a gateway node’s anti-entropy sweep needs to ask “give me everything since the epoch I last saw.” usage_rollups_minute is partitioned by day because it is the highest-volume table by far and because both the dashboard’s typical query window (last 24 hours to 30 days) and the retention policy (drop rollups after a year) align naturally with time-range partitioning.

A key’s lifecycle, traced in the diagram above: a row is born active in api_keys the moment the Key Issuance Service creates it, gets read (never written) by gateway nodes on every cache-miss refresh, and transitions to rotating or revoked exactly once, each transition appending a row to revocation_log that fans out over pub/sub before the anti-entropy sweep ever needs to look at it. A usage_rollups_minute row is born the instant the first event lands in a given minute bucket, is updated in place by the aggregator as more events arrive in that same minute, and becomes immutable once the minute closes.

Key Algorithms and Protocols

HMAC Key Hashing vs Password Hashing Algorithms

API keys are high-entropy, machine-generated secrets, unlike user passwords, so the algorithm choice differs from something like bcrypt or Argon2. A password hashing algorithm is deliberately slow to resist offline brute-force guessing against a low-entropy human-chosen secret; an API key already carries roughly 256 bits of randomness, so a fast cryptographic hash like HMAC-SHA256 with a server-side pepper is both sufficient and necessary, since a deliberately slow hash would add measurable latency multiplied across every authentication check on the hot path.

# Why HMAC-SHA256 and not bcrypt/Argon2 for API keys - a throughput comparison
# Demonstrates: the practical cost difference that makes hash choice a real design decision
import hashlib
import hmac
import time

PEPPER = b"server-side-secret"

def hmac_verify(candidate: str, stored_hash: bytes) -> float:
    start = time.perf_counter()
    hmac.new(PEPPER, candidate.encode(), hashlib.sha256).digest()
    return time.perf_counter() - start
    # Typical cost: low single-digit microseconds per call

# bcrypt/Argon2 deliberately cost 50-200+ milliseconds per call at a secure work
# factor, a cost that is the entire point for low-entropy passwords but would
# make 200,000 checks/second require tens of thousands of dedicated CPU cores

The property that makes this safe is that the pepper never lives alongside key_hash in the same datastore; an attacker who dumps the api_keys table gets hashes they still cannot invert without also compromising the separate secrets store holding the pepper. Brute-forcing a specific 256-bit secret directly is computationally infeasible regardless of hash speed, which is precisely why a fast hash is the correct tradeoff here and would be the wrong one for a human-chosen password.

Scope Authorization via Bitmask Sets

At small scale, storing scopes as a text array and checking membership with a linear scan is fine. At 5 million keys and 200,000 checks a second, packing scopes into a bitmask turns every check into a single integer AND.

# Scope bitmask encoding and O(1) authorization check
# Demonstrates: packing a small, fixed scope taxonomy into an integer bitmask
SCOPE_BITS = {
    "orders:read": 1 << 0,
    "orders:write": 1 << 1,
    "webhooks:manage": 1 << 2,
    "refunds:write": 1 << 3,
    "analytics:read": 1 << 4,
}

def encode_scopes(scope_names: list[str]) -> int:
    mask = 0
    for name in scope_names:
        mask |= SCOPE_BITS[name]
    return mask

def has_scope(granted_mask: int, required_scope: str) -> bool:
    required_bit = SCOPE_BITS[required_scope]
    return (granted_mask & required_bit) == required_bit

def has_all_scopes(granted_mask: int, required_scopes: list[str]) -> bool:
    required_mask = 0
    for scope in required_scopes:
        required_mask |= SCOPE_BITS[scope]
    return (granted_mask & required_mask) == required_mask

A bitmask check is O(1) regardless of how many scopes exist, versus O(n) for a linear scan through a text array or a set lookup with hashing overhead per call, and it packs an entire scope grant into a single integer that fits trivially in the gateway’s local cache entry alongside the rest of a key’s metadata. The tradeoff is a fixed, centrally defined scope taxonomy; a bitmask does not extend gracefully to arbitrary, tenant-defined scope strings the way a text array does, so platforms with a small number of well-known scopes are the right fit and platforms wanting fully dynamic per-tenant scope names should keep the text-array representation instead.

Revocation Epoch Comparison as a Conflict-Free Merge

The gateway’s Invalidate call and the anti-entropy reconciliation loop can both attempt to update the same cache entry concurrently, from different threads, in any order.

# Epoch-based last-writer-wins merge for concurrent cache invalidation
# Demonstrates: idempotent, order-independent application of revocation events
def merge_revocation(current_epoch: int, current_status: str, incoming_epoch: int, incoming_status: str) -> tuple[int, str]:
    # Applying the same event twice, or applying two events out of order, always
    # converges to the same final state: whichever epoch is numerically highest wins
    if incoming_epoch >= current_epoch:
        return incoming_epoch, incoming_status
    return current_epoch, current_status

This is a small, purpose-built instance of a last-writer-wins register, the same conflict-resolution idea behind CRDTs used in multi-master replicated datastores: applying the same update twice or applying two updates out of order both converge to an identical final state, which is exactly the property required when a pub/sub message and an anti-entropy sweep can race against each other on the same gateway node. The time and space cost is O(1) per key, a single integer comparison, which is why it can run on every cache write without adding measurable overhead.

Scaling and Performance

The Gateway Auth Middleware and the Per-Key Rate Limiter are both designed to scale horizontally with the gateway fleet itself: authentication is local-cache-first with no cross-node coordination, and rate-limit state lives in a sharded Redis cluster keyed by key identifier rather than in any single gateway node’s memory.

Capacity Estimation:

Given:
  - 200,000 requests/second at peak, 60,000/second sustained average
  - 5,000,000 active keys, ~150 gateway nodes across 3 regions
  - Local cache entry: ~300 bytes (scopes, rate config, status, epoch)
  - Usage event: ~120 bytes; 300M events/day at sustained average traffic
  - Revocation rate: ~50/minute steady state, spikes to 500/minute during an incident

Gateway local cache memory (per node, worst case all keys hot):
  5,000,000 keys * 300 bytes = 1.5 GB per node
  In practice each node only caches keys it actually sees traffic for,
  so real usage is a small fraction of this ceiling per region

Rate limiter throughput (Redis cluster, one round trip per request):
  200,000/s * 1 Lua script call = 200,000 ops/sec across the cluster
  A single Redis node handles roughly 80,000-100,000 simple ops/sec,
  so the cluster needs at least 3 shards at peak, sized to 6+ with headroom

Usage event pipeline (Kafka, async):
  300,000,000 events/day * 120 bytes = ~36 GB/day raw
  Aggregated into per-key-per-minute rollups: 5M keys * 1440 min/day
  worst case ~7.2B possible rows/day, but only keys with actual traffic
  produce a row, typically under 50M rollup rows/day in practice

Revocation fanout (pub/sub, worst case during an incident):
  500 revocations/minute * 150 subscribed gateway nodes
  = 75,000 message deliveries/minute = ~1,250/second, trivial for any
  pub/sub broker sized for this fleet

The dominant cost at peak is the rate limiter’s Redis round trip, not the authentication check itself, which is why the rate limiter cluster is sharded independently and sized with headroom above the gateway fleet’s own scaling. Read/write ratio on api_keys favors reads overwhelmingly (cache-miss refreshes only) while writes concentrate on usage_rollups_minute and revocation_log, both structured for high write throughput rather than point-lookup performance. The hottest keys across the whole system are, unsurprisingly, the platform’s highest-traffic integration partners, whose rate-limit buckets and cache entries get touched on nearly every request; pin those specific key shards to dedicated, over-provisioned Redis nodes rather than letting a handful of very hot keys degrade the shared cluster for everyone else.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Pub/sub broker outage	Subscriber heartbeat/connection alert on every gateway node	Revocations stop propagating via push; SLA at risk	Anti-entropy sweep against revocation_log becomes the primary path until the broker recovers; alert paging on sustained outage
Rate limiter Redis shard down	Connection errors/timeouts from the gateway’s rate-limit calls	Keys mapped to that shard cannot be rate-limited accurately	Fail toward a conservative default (deny burst above baseline, or route to a replica) rather than allowing unlimited traffic through
Control plane database (Postgres) unavailable	Health check and write-path error rate on the Key Issuance Service	New key issuance and explicit revocations blocked	Gateway nodes keep serving from local cache uninterrupted; reads are unaffected since the hot path never depends on a live DB connection
Gateway node cold start after restart or deploy	Elevated cache-miss rate and control-plane call volume from one node	Temporary latency increase and control-plane load spike for that node	Pre-warm from a recently-seen-keys snapshot; stagger fleet-wide restarts during deploys
Clock skew between gateway nodes and the control plane	Epoch comparisons behave inconsistently across nodes for the same event	A node with a skewed clock could delay applying or mis-order a revocation	NTP-sync all nodes; epoch is assigned once by the control plane at revocation time, not recomputed locally, so skew cannot corrupt the ordering itself
Usage event pipeline consumer lag	Kafka consumer-group lag metric on the aggregation job	Usage dashboards and billing numbers run stale	Backpressure and autoscale consumers; the bounded local buffer drops oldest events under sustained overload rather than blocking request threads

Comparison of Approaches

Approach	Revocation latency	Complexity	Failure mode	Best fit
Synchronous DB check per request	Immediate (always current)	Low	Database becomes the platform’s busiest, most fragile dependency at scale	Low-traffic internal tools where request volume never threatens the database
Cache with long TTL, no push invalidation	Bounded by TTL (minutes)	Low	Revoked keys keep working for the full TTL window; misses any sub-second SLA	Systems where a multi-minute revocation delay is genuinely acceptable
Self-contained tokens (JWT) with short expiry, no revocation list	Bounded by token expiry only	Medium	A compromised token works until it naturally expires; no way to force it dead sooner without a blocklist, which reintroduces the exact problem this avoids	Short-lived, low-privilege tokens where a few minutes of exposure is an acceptable risk
Push-based revocation fanout with local cache and TTL backstop (this design)	Sub-second, bounded by fanout latency	High (pub/sub infra, epoch handling, anti-entropy)	Broker outage delays propagation to the anti-entropy sweep’s slower cycle	High-traffic platforms where both hot-path latency and fast revocation are hard requirements simultaneously
Centralized rate-limit and auth check via a shared in-memory service (no gateway-local cache at all)	Immediate	Medium	The shared service becomes a single point of contention and a single point of failure for every request on the platform	Smaller fleets where a single well-provisioned service can absorb full request volume comfortably

We would pick the push-based fanout design for any platform where request volume is high enough that a synchronous per-request check is not viable and where revocation speed is a real security requirement, not a nice-to-have. The alternatives all trade one of the two constraints away entirely: a synchronous check trades away hot-path performance, a long TTL cache trades away revocation speed, and self-contained tokens trade away the ability to revoke at all before natural expiry. In practice, many platforms combine approaches: short-lived, self-contained tokens for browser sessions where a JWT’s tradeoffs are acceptable, and the push-based, long-lived API key model described here for server-to-server integrations where partners expect a credential to work indefinitely until explicitly rotated or revoked.

Key Takeaways

• Key hashing for storage means the platform stores only an HMAC of a key’s secret behind a server-side pepper, so a database breach alone never yields a usable credential.
• Scope-based authorization limits every key to exactly the resources and actions it was granted, checked as an O(1) bitmask operation on the hot path rather than a slower per-request lookup.
• Per-key rate limit counters, implemented as a sharded token bucket, guarantee one key’s traffic can never borrow capacity from or get penalized by any other key’s traffic.
• Revocation propagation through a push-based pub/sub fanout, backed by an anti-entropy sweep, is what actually achieves a sub-second revocation SLA, not a shorter cache TTL.
• Local cache with TTL keeps the hot path free of synchronous control-plane calls for the overwhelming majority of requests, with the TTL acting only as a self-healing backstop.
• The usage event pipeline is deliberately best-effort and fully decoupled from the authorization decision, so analytics accuracy is never purchased at the cost of request latency.
• Key rotation workflow models rotation as two coexisting active credentials sharing an identity during a grace window, never a synchronized, all-at-once swap.
• Revocation epochs, not boolean flags, make invalidation idempotent and order-independent, which is required the moment more than one delivery path (push and anti-entropy) can update the same cache entry.

The counter-intuitive lesson is that the hardest constraint in this system, a sub-second revocation guarantee, is not solved by making anything faster. It is solved by changing who initiates the update: instead of every gateway node repeatedly asking “has anything changed”, the control plane tells every node the instant something does, and the slow, periodic polling loop is demoted to a rarely-used backstop instead of the primary mechanism. That inversion, push instead of poll, is what lets the hot path stay entirely local while the system-wide state still converges in well under a second.

Frequently Asked Questions

Q: Why not just use short-lived JWTs instead of long-lived opaque API keys with a revocation system?

A: JWTs move the validation cost off the server (no lookup needed to check a signature) but they trade away exactly the revocation guarantee this system requires: a self-contained token is valid until it expires, full stop, and the only way to kill it sooner is a revocation blocklist, which reintroduces the same propagation problem this design solves directly. Long-lived, directly-issued API keys are also simply what B2B integration partners expect; nobody wants to implement an OAuth refresh flow just to call a webhook API.

Q: Why not just shrink the local cache TTL to a second or two instead of building a whole pub/sub propagation layer?

A: A 1-2 second TTL with no push mechanism means every gateway node refetches every actively-used key’s metadata from the control plane every 1-2 seconds, which at 5 million active keys and 150 nodes turns the control plane into the platform’s hottest, most contended dependency, the exact failure mode a synchronous per-request check produces. Push-based invalidation lets the TTL stay long (30-60 seconds) for efficiency while still reacting to an actual revocation in well under a second.

Q: How do you prevent a compromised admin credential from mass-revoking every key on the platform as a denial-of-service attack?

A: Revocation actions are authenticated and rate-limited themselves, logged with an actor_id in revocation_log for full audit traceability, and a sustained spike in revocation volume (the “500/minute during an incident” case in the capacity math) triggers an anomaly alert distinct from the normal incident-response revocation flow. This is a real operational risk that argues for requiring elevated, logged authorization on the revocation endpoint itself, separate from ordinary key-management permissions.

Q: Why hash the key with HMAC instead of a slow password-hashing algorithm like bcrypt or Argon2?

A: Password hashing algorithms are deliberately slow to compensate for the low entropy of human-chosen secrets; an API key already carries around 256 bits of cryptographically random entropy, so there is no offline-brute-force weakness to compensate for. Running a deliberately slow hash 200,000 times a second would require dedicating enormous compute purely to authentication with no corresponding security benefit.

Q: What stops the usage event pipeline’s async nature from silently under-billing a customer?

A: The bounded local buffer only drops events under genuinely extreme, sustained overload, which is treated as an accepted tradeoff and is monitored as its own metric (buffer-full drop rate) rather than a silent failure. For platforms where exact billing precision matters more than hot-path latency, the buffer can be backed by a local durable queue (write-ahead to disk before the async flush) at the cost of a small amount of added latency, a tuning knob rather than a fixed design constraint.

Q: Why does key rotation create two separate rows instead of just updating the secret on the existing key row?

A: Both the old and new secret need to authenticate successfully and independently during the grace window, and each needs its own key_hash for lookup; a single row can only store one current hash at a time. Modeling rotation as a rotated_from link between two rows also gives a clean, queryable audit trail of a key’s rotation history, which an in-place secret swap would destroy the moment the update happened.

Interview Questions

Q: Design the revocation propagation mechanism so it meets a 1-second SLA across 150 gateway nodes spread across three regions, and explain what happens when the pub/sub broker itself is unavailable.

Expected depth: Discuss push-based fanout over poll-based checking, why a monotonic epoch rather than a boolean status field is required under concurrent, out-of-order delivery, and the anti-entropy sweep as a backstop when the primary broker path is degraded. Cover cross-region replication lag as a real constraint on the SLA and how a regional broker outage should degrade gracefully rather than silently missing the SLA everywhere.

Q: Walk through how you would shard the per-key rate limiter across a Redis cluster and what happens when you need to add capacity during a traffic spike.

Expected depth: Cover consistent hashing to minimize remapping when the shard count changes, why a naive hash(key) % node_count scheme would be dangerous specifically during a scale-up event, and how to identify and specially provision for a small number of extremely hot keys (the platform’s biggest integration partners) rather than assuming uniform load across all 5 million keys.

Q: A customer reports that their key stopped working immediately after they rotated it, even though your rotation workflow promises a grace period. How do you debug this?

Expected depth: Discuss checking whether the old key’s row actually transitioned to rotating rather than revoked at rotation time, whether the gateway’s local cache picked up a stale revoked status from a race with an unrelated event, and whether the rotation grace period sweep job incorrectly matched this key. Tie this back to the epoch-based merge logic and how a bug there could revoke a key prematurely if epochs are generated or compared incorrectly.

Q: How would you extend this system to detect and auto-revoke a key that has clearly leaked, for example one committed to a public GitHub repository, without waiting for the customer to notice?

Expected depth: Discuss integrating with a leaked-secret scanning signal (matching the key’s non-secret prefix pattern against public commit scanners), automatically transitioning a matched key to revoked with reason = 'leak_detected', and notifying the owner immediately rather than silently killing their integration. Cover the tradeoff of false positives here (a prefix match is not proof of an actual working leaked secret) and why a fast, reversible revoke-and-notify flow is safer than a slower manual review step for this specific failure mode.

Q: How do you scale usage analytics so a customer’s real-time dashboard can show accurate per-key request counts without querying 300 million raw events per day?

Expected depth: Discuss the pre-aggregation strategy (per-key, per-minute rollups written by a stream aggregator, not computed on read), partitioning the rollup table by time for cheap retention and query performance, and the tradeoff between rollup granularity (minute versus hour) and how quickly a dashboard reflects very recent traffic. Mention that raw events can still be retained separately in cheaper storage for occasional deep debugging without that retention affecting the dashboard’s hot query path.