Think of every LLM request as a taxi ride with no meter. A tenant sends a prompt, a GPU spins up, and the cost accumulates token by token until generation finishes. By the time you know the bill, the money is gone. For a single request that is fine. For a multi-tenant platform handling 10,000 requests per second with tenants running 128,000-token context windows at $0.06 per million output tokens, the absence of admission control converts a software bug into a $50,000 monthly overrun before the on-call engineer gets paged.

The naive approach is to track usage after the fact and cut tenants off when they exceed their quota. That approach has a fatal flaw: the GPU does not care about your accounting job. Output tokens generated by over-budget requests are real silicon cycles, real cooling watts, and real dollars billed by your cloud provider. Post-hoc accounting is a refund process dressed up as a quota system.

Correct token budget management works like a prepaid phone plan, not a credit card. Before a single GPU cycle fires, a pre-flight check verifies that the tenant has enough tokens remaining, reserves the estimated cost atomically, and either admits or rejects the request in under 2 milliseconds. Over-budget tenants hit a 429 before inference starts. GPU time goes to tenants who can pay for it.

This post designs that system end to end: the enforcement layer that runs before every request, the token counter that knows exactly how much each model will consume, the quota store that handles 10,000 concurrent atomic decrement operations in Redis, and the billing aggregator that makes the numbers match the cloud invoice.

Requirements and Constraints

Functional Requirements

• Admission control: Reject requests from tenants who have exhausted their token budget before inference starts, returning a structured 429 with retry timing.
• Real-time token counting: Count input tokens accurately per model using the correct tokenizer (cl100k_base for GPT-4-class models, a model-specific tokenizer for open-weight models) before admission.
• Output token tracking: Accumulate output tokens during streaming generation and attribute them to the originating tenant in real time.
• Budget windows: Enforce budgets at per-minute, per-day, and per-month granularities simultaneously, with configurable soft and hard limits per window.
• Cost attribution: Track input and output tokens separately because they carry different prices per model tier, and attribute costs to specific models and API keys.
• Alerts and suspension: Fire webhook alerts at configurable soft-limit thresholds (80 percent, 95 percent), suspend tenants automatically at 100 percent of the hard limit, and allow a configurable grace buffer for in-flight requests.

Non-Functional Requirements

Metric	Target	Notes
Admission control latency	under 2ms p99	Added overhead per request on the critical path
Throughput	10,000 req/s	Across all tenants simultaneously
Token count accuracy	within 2% of actual billing	tiktoken produces exact counts for OpenAI models
Counter drift	under 0.5% across budget windows	Redis atomic ops prevent split-brain; audit job detects drift
Alert delivery latency	under 5 seconds	Webhook with async queue, not blocking the request path
Quota enforcement latency	under 2ms p99	Redis Lua script evaluated atomically per request

Constraints

• No GPU waste on rejected requests. The enforcement check must complete before any model is loaded or KV cache allocated.
• Atomic decrement or reject. If two pods check a budget simultaneously and both see “budget available,” only one must win. This requires a compare-and-decrement atomic operation, not a read-then-write.
• Streaming complicates output accounting. We do not know how many output tokens a streaming request will produce until the stream ends. We must pessimistically reserve max_tokens at admission and refund the delta on completion.
• Input and output tokens price differently. A typical 70B-class model charges $0.03 per million input tokens and $0.06 per million output tokens. A budget system that treats them as the same unit will systematically undercount cost.
• Tokenizer per model. GPT-4o uses cl100k_base. Claude uses a different tokenizer. Llama 3 uses tiktoken with a SentencePiece vocabulary. Token counts are not portable across model families.

High-Level Architecture

The system breaks into four planes that operate at different latency tiers.

Enforcement Plane runs synchronously on every request path. The API Gateway calls the Budget Enforcement Layer, which counts input tokens, performs an atomic check-and-decrement in Redis, and either issues an admit decision or returns a 429. The entire round trip must complete in under 2ms.

Data Plane handles the inference itself and the real-time tracking of output tokens during streaming. The Streaming Token Counter sits between the LLM inference engine and the gateway, counting tokens as they arrive in the SSE stream and accumulating them in Redis.

Billing Plane operates asynchronously. The Billing Aggregator reads usage events from a queue, normalizes them to cost, and flushes to PostgreSQL in batches. It reconciles Redis counters against actual model billing logs to detect drift.

Alert Plane monitors thresholds and fires webhooks. It reads budget utilization from Redis on a configurable interval (typically 15 seconds) and delivers soft-limit warnings before tenants hit the hard cutoff.

The end-to-end flow for an admitted request:

1. Client sends request to API Gateway.
2. Gateway calls Budget Enforcement Layer with tenant ID, model, input text, and max_tokens.
3. Enforcement Layer tokenizes the input, adds max_tokens for a pessimistic estimate, and calls EVAL on the Redis Lua script to atomically check and decrement.
4. If admitted, the request proceeds to inference. If rejected, a 429 is returned immediately.
5. During generation, the Streaming Token Counter tallies output tokens per SSE chunk and writes them to a Redis string with INCRBY.
6. On stream end, the Enforcement Layer refunds max_tokens - actual_output_tokens back to the per-minute bucket.
7. A usage event is published to the message queue for the Billing Aggregator.
8. The Billing Aggregator batches events and writes to PostgreSQL every 5 seconds.

The Budget Enforcement Layer

The Enforcement Layer is a FastAPI service that runs synchronously on the request path. Its job is to accept or reject requests in under 2ms based on current budget state. It must handle 10,000 requests per second, which means each instance handles approximately 1,000 req/s on a 10-pod deployment.

Pre-flight Admission Control

from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse
import redis.asyncio as redis
import tiktoken
import time

app = FastAPI()
redis_client = redis.Redis(host="redis-cluster", port=6379, decode_responses=True)

# Pre-load tokenizers at startup, not per-request
TOKENIZERS = {
    "gpt-4o": tiktoken.get_encoding("cl100k_base"),
    "gpt-4o-mini": tiktoken.get_encoding("cl100k_base"),
    "gpt-4-turbo": tiktoken.get_encoding("cl100k_base"),
}

def count_tokens(model: str, text: str) -> int:
    enc = TOKENIZERS.get(model)
    if enc is None:
        # Fallback: approximate 4 chars per token for unknown models
        return max(1, len(text) // 4)
    return len(enc.encode(text))

@app.post("/v1/admit")
async def admit_request(request: Request):
    body = await request.json()
    tenant_id: str = body["tenant_id"]
    model: str = body["model"]
    prompt: str = body["prompt"]
    max_tokens: int = body.get("max_tokens", 1024)

    input_tokens = count_tokens(model, prompt)
    # Pessimistic reservation: full input + worst-case output
    reserved_tokens = input_tokens + max_tokens

    admitted, reason = await atomic_check_and_decrement(
        tenant_id=tenant_id,
        model=model,
        input_tokens=input_tokens,
        reserved_tokens=reserved_tokens,
    )

    if not admitted:
        raise HTTPException(
            status_code=429,
            detail={
                "error": "token_budget_exceeded",
                "reason": reason,
                "tenant_id": tenant_id,
                "retry_after_seconds": await seconds_until_window_reset(tenant_id),
            },
        )

    return {
        "admitted": True,
        "input_tokens": input_tokens,
        "reserved_tokens": reserved_tokens,
        "admission_id": f"{tenant_id}:{int(time.time() * 1000)}",
    }

Redis Lua Script for Atomic Check-and-Decrement

The core of admission control is a Lua script that Redis evaluates as a single atomic transaction. No other command from any client runs between the reads and the decrements.

-- atomic_admit.lua
-- KEYS[1] = per-minute counter key  e.g. quota:tenant123:tokens:minute:2026072914
-- KEYS[2] = per-day counter key      e.g. quota:tenant123:tokens:day:20260729
-- KEYS[3] = per-month counter key    e.g. quota:tenant123:tokens:month:202607
-- KEYS[4] = tenant config hash key   e.g. quota:tenant123:config
-- ARGV[1] = reserved_tokens (input + max_output)
-- ARGV[2] = current unix timestamp (for TTL initialization)
-- Returns: {"ok", input_tokens_used} or {"err", reason}

local reserved  = tonumber(ARGV[1])
local now       = tonumber(ARGV[2])

-- Load tenant config
local minute_limit = tonumber(redis.call('HGET', KEYS[4], 'minute_limit') or 0)
local day_limit    = tonumber(redis.call('HGET', KEYS[4], 'day_limit')    or 0)
local month_limit  = tonumber(redis.call('HGET', KEYS[4], 'month_limit')  or 0)
local suspended    = redis.call('HGET', KEYS[4], 'suspended')

if suspended == '1' then
    return {'err', 'account_suspended'}
end

-- Read current counters (GETEX would be cleaner but KEYS[n] pattern needs GET)
local used_minute = tonumber(redis.call('GET', KEYS[1]) or 0)
local used_day    = tonumber(redis.call('GET', KEYS[2]) or 0)
local used_month  = tonumber(redis.call('GET', KEYS[3]) or 0)

-- Check all three windows before touching any counter
if minute_limit > 0 and (used_minute + reserved) > minute_limit then
    return {'err', 'minute_limit_exceeded'}
end
if day_limit > 0 and (used_day + reserved) > day_limit then
    return {'err', 'day_limit_exceeded'}
end
if month_limit > 0 and (used_month + reserved) > month_limit then
    return {'err', 'month_limit_exceeded'}
end

-- All checks passed; increment all three windows atomically
local new_minute = redis.call('INCRBY', KEYS[1], reserved)
local new_day    = redis.call('INCRBY', KEYS[2], reserved)
local new_month  = redis.call('INCRBY', KEYS[3], reserved)

-- Set TTL only if the key was just created (INCRBY on a missing key starts at 0)
-- 90 seconds for minute window, 26 hours for day, 35 days for month
if new_minute == reserved then redis.call('EXPIRE', KEYS[1], 90) end
if new_day    == reserved then redis.call('EXPIRE', KEYS[2], 93600) end
if new_month  == reserved then redis.call('EXPIRE', KEYS[3], 3024000) end

return {'ok', tostring(new_minute), tostring(new_day), tostring(new_month)}

# Calling the Lua script from Python
LUA_SCRIPT = open("atomic_admit.lua").read()
_admit_script = None

async def get_admit_script():
    global _admit_script
    if _admit_script is None:
        _admit_script = redis_client.register_script(LUA_SCRIPT)
    return _admit_script

async def atomic_check_and_decrement(
    tenant_id: str,
    model: str,
    input_tokens: int,
    reserved_tokens: int,
) -> tuple[bool, str]:
    now = int(time.time())
    minute_key = f"quota:{tenant_id}:tokens:minute:{now // 60}"
    day_key    = f"quota:{tenant_id}:tokens:day:{now // 86400}"
    month_key  = f"quota:{tenant_id}:tokens:month:{now // 2592000}"
    config_key = f"quota:{tenant_id}:config"

    script = await get_admit_script()
    result = await script(
        keys=[minute_key, day_key, month_key, config_key],
        args=[str(reserved_tokens), str(now)],
    )
    status = result[0]
    if status == "ok":
        return True, ""
    return False, result[1]

The Token Counter

Accurate token counting is the load-bearing piece of the whole system. An off-by-10-percent count means a tenant can send 10 percent more compute than they paid for, or get rejected 10 percent sooner than they should. The tokenizer must match the model exactly.

tiktoken Integration and Per-Model Tokenizers

import tiktoken
from functools import lru_cache

# Encoding names by model family
MODEL_ENCODING_MAP = {
    # GPT-4 family: cl100k_base
    "gpt-4o":               "cl100k_base",
    "gpt-4o-mini":          "cl100k_base",
    "gpt-4-turbo":          "cl100k_base",
    "gpt-4":                "cl100k_base",
    "gpt-3.5-turbo":        "cl100k_base",
    # GPT-3 legacy
    "text-davinci-003":     "p50k_base",
    # Codex
    "code-davinci-002":     "p50k_base",
}

@lru_cache(maxsize=16)
def get_encoding(encoding_name: str) -> tiktoken.Encoding:
    """Cache encodings - loading from disk takes ~40ms the first time."""
    return tiktoken.get_encoding(encoding_name)

def count_chat_tokens(model: str, messages: list[dict]) -> int:
    """
    Count tokens for a chat completion request.
    Includes per-message overhead (4 tokens per message for GPT-4-class models).
    Reference: https://platform.openai.com/docs/guides/chat/managing-tokens
    """
    encoding_name = MODEL_ENCODING_MAP.get(model, "cl100k_base")
    enc = get_encoding(encoding_name)

    tokens_per_message = 4   # every message: <|im_start|>{role}\n{content}<|im_end|>\n
    tokens_per_name = -1     # if there is a name, the role is omitted

    total = 0
    for msg in messages:
        total += tokens_per_message
        for key, value in msg.items():
            total += len(enc.encode(str(value)))
            if key == "name":
                total += tokens_per_name
    total += 3  # every reply is primed with <|im_start|>assistant
    return total

def count_completion_tokens(model: str, text: str) -> int:
    """Count tokens for a raw completion (non-chat) prompt."""
    encoding_name = MODEL_ENCODING_MAP.get(model, "cl100k_base")
    enc = get_encoding(encoding_name)
    return len(enc.encode(text))

# For open-weight models served via vLLM, use the model's own tokenizer
from transformers import AutoTokenizer
import asyncio

_hf_tokenizers: dict[str, AutoTokenizer] = {}

async def count_tokens_vllm(model_path: str, text: str) -> int:
    """
    Count tokens using the HuggingFace tokenizer for open-weight models.
    Load once and cache. Llama 3 uses a different BPE vocabulary than cl100k_base.
    """
    if model_path not in _hf_tokenizers:
        loop = asyncio.get_event_loop()
        tokenizer = await loop.run_in_executor(
            None, AutoTokenizer.from_pretrained, model_path
        )
        _hf_tokenizers[model_path] = tokenizer
    enc = _hf_tokenizers[model_path]
    return len(enc.encode(text, add_special_tokens=False))

Streaming Token Accumulation During SSE

Output token counting cannot wait for the stream to finish. We need a running count as each chunk arrives, both for the refund calculation and for soft-limit warnings during generation.

import asyncio
import httpx
import redis.asyncio as redis
from dataclasses import dataclass, field

@dataclass
class StreamAccumulator:
    tenant_id: str
    admission_id: str
    model: str
    reserved_tokens: int
    input_tokens: int
    output_tokens: int = 0
    chunks: list[str] = field(default_factory=list)

    def add_chunk(self, text: str, token_count: int) -> None:
        self.output_tokens += token_count
        self.chunks.append(text)

    @property
    def total_tokens(self) -> int:
        return self.input_tokens + self.output_tokens

async def stream_and_count(
    upstream_url: str,
    request_body: dict,
    accumulator: StreamAccumulator,
    redis_client: redis.Redis,
) -> None:
    """
    Proxy the SSE stream from the LLM, count tokens per chunk,
    and maintain a live counter in Redis.
    """
    encoding_name = MODEL_ENCODING_MAP.get(accumulator.model, "cl100k_base")
    enc = get_encoding(encoding_name)

    live_key = f"quota:{accumulator.tenant_id}:live:{accumulator.admission_id}"

    async with httpx.AsyncClient(timeout=120.0) as client:
        async with client.stream("POST", upstream_url, json=request_body) as response:
            async for line in response.aiter_lines():
                if not line.startswith("data:"):
                    continue
                data = line[5:].strip()
                if data == "[DONE]":
                    break

                import json
                chunk_data = json.loads(data)
                delta = chunk_data.get("choices", [{}])[0].get("delta", {})
                content = delta.get("content", "")
                if content:
                    chunk_tokens = len(enc.encode(content))
                    accumulator.add_chunk(content, chunk_tokens)
                    # Increment live counter in Redis for soft-limit monitoring
                    await redis_client.incrby(live_key, chunk_tokens)
                    yield line + "\n\n"

    # Stream ended: refund unused reserved output tokens
    actual_output = accumulator.output_tokens
    over_reserved = accumulator.reserved_tokens - accumulator.input_tokens - actual_output
    if over_reserved > 0:
        now = int(asyncio.get_event_loop().time())
        minute_key = f"quota:{accumulator.tenant_id}:tokens:minute:{int(now) // 60}"
        day_key    = f"quota:{accumulator.tenant_id}:tokens:day:{int(now) // 86400}"
        month_key  = f"quota:{accumulator.tenant_id}:tokens:month:{int(now) // 2592000}"
        pipe = redis_client.pipeline()
        pipe.decrby(minute_key, over_reserved)
        pipe.decrby(day_key, over_reserved)
        pipe.decrby(month_key, over_reserved)
        pipe.delete(live_key)
        await pipe.execute()

The Quota Store

The Quota Store is the Redis tier that holds budget state for all tenants. Its design determines throughput, correctness, and failure behavior under load.

Redis Data Structures and Key Schema

We use three types of Redis data structures for three different purposes.

String keys with TTL for fixed-window counters. Simple, fast, and self-expiring. The key encodes the window boundary so we never need a separate cleanup job.

quota:{tenant_id}:tokens:minute:{unix_minute}   -> integer (tokens used)
quota:{tenant_id}:tokens:day:{unix_day}         -> integer (tokens used)
quota:{tenant_id}:tokens:month:{unix_month}     -> integer (tokens used)

Hash keys for tenant configuration. One HGETALL retrieves the full quota config in a single round trip.

quota:{tenant_id}:config -> {
    minute_limit:    "2000000",    # 2M tokens/minute
    day_limit:       "50000000",   # 50M tokens/day
    month_limit:     "1000000000", # 1B tokens/month
    soft_pct:        "80",         # alert at 80%
    hard_pct:        "100",        # hard cutoff at 100%
    grace_tokens:    "10000",      # extra tokens for in-flight at cutoff
    suspended:       "0",          # 1 = suspended
    plan:            "pro",
    model_tier:      "all",        # "small", "large", "all"
}

Sorted Sets for sliding window implementation. Score is the unix timestamp, member is the usage event ID. We use ZRANGEBYSCORE with the window lower bound to sum only events within the window.

quota:{tenant_id}:sliding:tokens -> sorted set, score=timestamp, member="{event_id}:{token_count}"

Sliding Window vs Fixed Window

Fixed windows are fast and storage-efficient but have a burst edge case: a tenant can consume their full minute quota in the last second of one window, then immediately consume it again in the first second of the next window, effectively consuming 2x the quota in 2 seconds.

import time
import redis.asyncio as redis

async def sliding_window_count(
    redis_client: redis.Redis,
    tenant_id: str,
    window_seconds: int = 60,
) -> int:
    """
    Sum tokens used in the last `window_seconds` using a sorted set.
    Each member encodes the token count: "{event_id}:{tokens}".
    """
    now = time.time()
    window_start = now - window_seconds
    key = f"quota:{tenant_id}:sliding:tokens"

    # Remove expired members, then fetch the remaining range
    pipe = redis_client.pipeline()
    pipe.zremrangebyscore(key, "-inf", window_start)
    pipe.zrangebyscore(key, window_start, "+inf")
    _, members = await pipe.execute()

    total = 0
    for member in members:
        parts = member.rsplit(":", 1)
        if len(parts) == 2:
            total += int(parts[1])
    return total

async def sliding_window_add(
    redis_client: redis.Redis,
    tenant_id: str,
    event_id: str,
    tokens: int,
    ttl_seconds: int = 90,
) -> None:
    """Add a usage event to the sliding window sorted set."""
    key = f"quota:{tenant_id}:sliding:tokens"
    now = time.time()
    await redis_client.zadd(key, {f"{event_id}:{tokens}": now})
    await redis_client.expire(key, ttl_seconds)

For the per-minute admission check we use fixed windows in the Lua script because they are one INCRBY per key, not a ZRANGEBYSCORE over potentially thousands of members. Sliding windows serve the alerting path where a 200ms latency is acceptable and burst detection matters more than admission latency.

TTL-Based Window Management

Fixed-window keys expire automatically, which means there is no cleanup job to run. The TTL must be set only once per window boundary - the Lua script handles this by checking whether the counter value after INCRBY equals the increment amount (meaning the key was just created).

# TTL values per window type
WINDOW_TTLS = {
    "minute": 90,        # 60s window + 30s grace for in-flight requests
    "day":    93600,     # 86400s + 7200s grace
    "month":  3024000,   # 30 days + 5 days grace
}

# Key generation with time-boundary encoding
def make_window_keys(tenant_id: str, now: float) -> dict[str, str]:
    ts = int(now)
    return {
        "minute": f"quota:{tenant_id}:tokens:minute:{ts // 60}",
        "day":    f"quota:{tenant_id}:tokens:day:{ts // 86400}",
        "month":  f"quota:{tenant_id}:tokens:month:{ts // 2592000}",
        "config": f"quota:{tenant_id}:config",
    }

The Billing Aggregator

The Billing Aggregator is intentionally off the critical path. It reads usage events from a Kafka topic, normalizes them to dollar cost, and writes to PostgreSQL in batches. Its job is to make the numbers correct and auditable, not to be fast.

Async Flush to PostgreSQL

import asyncpg
import asyncio
import json
from dataclasses import dataclass
from datetime import datetime, timezone

# Pricing table: (input_price_per_million, output_price_per_million)
MODEL_PRICING = {
    "gpt-4o":           (5.00,  15.00),
    "gpt-4o-mini":      (0.15,   0.60),
    "gpt-4-turbo":      (10.00, 30.00),
    "meta-llama-3-70b": (0.59,   0.79),
    "meta-llama-3-8b":  (0.05,   0.08),
}

@dataclass
class UsageEvent:
    tenant_id: str
    api_key_id: str
    model: str
    input_tokens: int
    output_tokens: int
    admission_id: str
    started_at: datetime
    completed_at: datetime

    @property
    def input_cost_usd(self) -> float:
        input_price, _ = MODEL_PRICING.get(self.model, (0.03, 0.06))
        return (self.input_tokens / 1_000_000) * input_price

    @property
    def output_cost_usd(self) -> float:
        _, output_price = MODEL_PRICING.get(self.model, (0.03, 0.06))
        return (self.output_tokens / 1_000_000) * output_price

    @property
    def total_cost_usd(self) -> float:
        return self.input_cost_usd + self.output_cost_usd

class BillingAggregator:
    def __init__(self, dsn: str, batch_size: int = 500, flush_interval_s: float = 5.0):
        self.dsn = dsn
        self.batch_size = batch_size
        self.flush_interval_s = flush_interval_s
        self._buffer: list[UsageEvent] = []
        self._lock = asyncio.Lock()
        self._pool: asyncpg.Pool | None = None

    async def start(self):
        self._pool = await asyncpg.create_pool(self.dsn, min_size=2, max_size=10)
        asyncio.create_task(self._flush_loop())

    async def record(self, event: UsageEvent) -> None:
        async with self._lock:
            self._buffer.append(event)
            if len(self._buffer) >= self.batch_size:
                await self._flush()

    async def _flush_loop(self) -> None:
        while True:
            await asyncio.sleep(self.flush_interval_s)
            async with self._lock:
                if self._buffer:
                    await self._flush()

    async def _flush(self) -> None:
        if not self._buffer or self._pool is None:
            return
        batch = self._buffer[:]
        self._buffer.clear()

        rows = [
            (
                e.tenant_id, e.api_key_id, e.model,
                e.input_tokens, e.output_tokens,
                e.input_cost_usd, e.output_cost_usd, e.total_cost_usd,
                e.admission_id, e.started_at, e.completed_at,
            )
            for e in batch
        ]

        async with self._pool.acquire() as conn:
            await conn.executemany(
                """
                INSERT INTO usage_events (
                    tenant_id, api_key_id, model,
                    input_tokens, output_tokens,
                    input_cost_usd, output_cost_usd, total_cost_usd,
                    admission_id, started_at, completed_at
                ) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)
                ON CONFLICT (admission_id) DO NOTHING
                """,
                rows,
            )

Cost Attribution by Model and Tenant

The aggregator exposes a materialized view that the billing dashboard reads directly, rather than aggregating on every request.

-- Materialized view refreshed every 5 minutes
CREATE MATERIALIZED VIEW tenant_cost_summary AS
SELECT
    tenant_id,
    model,
    date_trunc('hour', started_at) AS hour,
    SUM(input_tokens)              AS total_input_tokens,
    SUM(output_tokens)             AS total_output_tokens,
    SUM(input_cost_usd)            AS total_input_cost_usd,
    SUM(output_cost_usd)           AS total_output_cost_usd,
    SUM(total_cost_usd)            AS total_cost_usd,
    COUNT(*)                       AS request_count
FROM usage_events
WHERE started_at >= NOW() - INTERVAL '32 days'
GROUP BY tenant_id, model, date_trunc('hour', started_at)
WITH DATA;

CREATE UNIQUE INDEX ON tenant_cost_summary (tenant_id, model, hour);

-- Refresh on a schedule
SELECT cron.schedule('*/5 * * * *', 'REFRESH MATERIALIZED VIEW CONCURRENTLY tenant_cost_summary');

Data Model

SQL DDL

-- Tenant table
CREATE TABLE tenants (
    id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name         TEXT NOT NULL,
    plan         TEXT NOT NULL DEFAULT 'free',   -- free, pro, enterprise
    suspended    BOOLEAN NOT NULL DEFAULT false,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Quota configuration per tenant
CREATE TABLE tenant_quotas (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id        UUID NOT NULL REFERENCES tenants(id),
    minute_limit     BIGINT NOT NULL DEFAULT 0,       -- 0 = unlimited
    day_limit        BIGINT NOT NULL DEFAULT 0,
    month_limit      BIGINT NOT NULL DEFAULT 0,
    soft_limit_pct   INT    NOT NULL DEFAULT 80,      -- alert threshold
    grace_tokens     BIGINT NOT NULL DEFAULT 10000,   -- in-flight buffer
    model_tier       TEXT   NOT NULL DEFAULT 'all',   -- all, small, large
    effective_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE (tenant_id)
);

-- Per-request usage events (append-only)
CREATE TABLE usage_events (
    id              BIGSERIAL PRIMARY KEY,
    tenant_id       UUID NOT NULL,
    api_key_id      UUID NOT NULL,
    model           TEXT NOT NULL,
    input_tokens    INT  NOT NULL,
    output_tokens   INT  NOT NULL,
    input_cost_usd  NUMERIC(12,8) NOT NULL,
    output_cost_usd NUMERIC(12,8) NOT NULL,
    total_cost_usd  NUMERIC(12,8) NOT NULL,
    admission_id    TEXT NOT NULL,
    started_at      TIMESTAMPTZ NOT NULL,
    completed_at    TIMESTAMPTZ NOT NULL,
    CONSTRAINT usage_events_admission_id_uniq UNIQUE (admission_id)
) PARTITION BY RANGE (started_at);

-- Monthly partitions for retention management
CREATE TABLE usage_events_2026_07 PARTITION OF usage_events
    FOR VALUES FROM ('2026-07-01') TO ('2026-08-01');
CREATE TABLE usage_events_2026_08 PARTITION OF usage_events
    FOR VALUES FROM ('2026-08-01') TO ('2026-09-01');

-- Budget alerts log
CREATE TABLE budget_alerts (
    id            BIGSERIAL PRIMARY KEY,
    tenant_id     UUID NOT NULL,
    alert_type    TEXT NOT NULL,     -- soft_limit, hard_limit, suspended
    window        TEXT NOT NULL,     -- minute, day, month
    used_tokens   BIGINT NOT NULL,
    limit_tokens  BIGINT NOT NULL,
    pct_used      NUMERIC(5,2) NOT NULL,
    fired_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    delivered_at  TIMESTAMPTZ
);

-- Indexes for dashboard queries
CREATE INDEX ON usage_events (tenant_id, started_at DESC);
CREATE INDEX ON usage_events (model, started_at DESC);
CREATE INDEX ON budget_alerts (tenant_id, fired_at DESC);

Redis Key Schema

# Fixed window counters (STRING, integer value, auto-expiring)
quota:{tenant_id}:tokens:minute:{unix_ts // 60}
quota:{tenant_id}:tokens:day:{unix_ts // 86400}
quota:{tenant_id}:tokens:month:{unix_ts // 2592000}

# Tenant config (HASH)
quota:{tenant_id}:config

# Live stream counter for in-progress requests (STRING, auto-expiring 300s)
quota:{tenant_id}:live:{admission_id}

# Sliding window for burst detection (SORTED SET, score=timestamp)
quota:{tenant_id}:sliding:tokens

# Suspension flag (STRING, "1" or absent)
quota:{tenant_id}:suspended

Protobuf for Usage Events

syntax = "proto3";
package tokenbudget.v1;

message UsageEvent {
    string tenant_id      = 1;
    string api_key_id     = 2;
    string model          = 3;
    int64  input_tokens   = 4;
    int64  output_tokens  = 5;
    string admission_id   = 6;
    int64  started_at_ms  = 7;   // unix milliseconds
    int64  completed_at_ms = 8;
    // Set by billing aggregator, not gateway
    double input_cost_usd  = 9;
    double output_cost_usd = 10;
}

message AdmissionResult {
    bool   admitted        = 1;
    string rejection_reason = 2;  // "minute_limit_exceeded" etc.
    int64  retry_after_s   = 3;
    int64  reserved_tokens = 4;
    string admission_id    = 5;
}

Key Algorithms and Protocols

Token Bucket Algorithm for Rate Limiting

The token bucket algorithm models a bucket that refills at a constant rate and is depleted by each request. It naturally handles bursts up to the bucket capacity while enforcing a steady-state throughput ceiling.

import time
import redis.asyncio as redis

class TokenBucket:
    """
    Token bucket for per-tenant request rate limiting.
    Separate from the token-count budget: this limits requests/second, not tokens.
    Implemented with Redis so it works across multiple gateway pods.
    """

    def __init__(
        self,
        redis_client: redis.Redis,
        capacity: int,          # max burst size in requests
        refill_rate: float,     # requests per second
    ):
        self.r = redis_client
        self.capacity = capacity
        self.refill_rate = refill_rate

    async def acquire(self, tenant_id: str, cost: int = 1) -> tuple[bool, float]:
        """
        Try to acquire `cost` tokens from the bucket.
        Returns (allowed, retry_after_seconds).
        """
        key_tokens = f"ratelimit:{tenant_id}:tokens"
        key_ts     = f"ratelimit:{tenant_id}:last_ts"
        now = time.monotonic()

        # Lua: refill bucket based on elapsed time, then check capacity
        lua = """
        local tokens_key = KEYS[1]
        local ts_key     = KEYS[2]
        local now        = tonumber(ARGV[1])
        local capacity   = tonumber(ARGV[2])
        local rate       = tonumber(ARGV[3])
        local cost       = tonumber(ARGV[4])

        local last_ts = tonumber(redis.call('GET', ts_key) or now)
        local elapsed = math.max(0, now - last_ts)
        local stored  = tonumber(redis.call('GET', tokens_key) or capacity)
        local refilled = math.min(capacity, stored + elapsed * rate)

        if refilled < cost then
            -- Not enough tokens; compute wait
            local deficit = cost - refilled
            local wait_s  = deficit / rate
            return {0, tostring(wait_s)}
        end

        redis.call('SET', tokens_key, tostring(refilled - cost), 'EX', 120)
        redis.call('SET', ts_key, tostring(now), 'EX', 120)
        return {1, '0'}
        """

        script = self.r.register_script(lua)
        result = await script(
            keys=[key_tokens, key_ts],
            args=[str(now), str(self.capacity), str(self.refill_rate), str(cost)],
        )
        allowed = result[0] == 1
        retry_after = float(result[1])
        return allowed, retry_after

Sliding Window Counter for Burst Detection

async def get_sliding_window_usage(
    redis_client: redis.Redis,
    tenant_id: str,
    window_seconds: int,
) -> dict[str, int]:
    """
    Compute token usage in a rolling window for the alerting path.
    More accurate than fixed windows at window boundaries.
    """
    now = time.time()
    window_start = now - window_seconds
    key = f"quota:{tenant_id}:sliding:tokens"

    pipe = redis_client.pipeline(transaction=False)
    pipe.zremrangebyscore(key, "-inf", window_start)
    pipe.zrangebyscore(key, window_start, "+inf", withscores=False)
    results = await pipe.execute()

    total_tokens = 0
    for member in results[1]:
        try:
            token_count = int(member.rsplit(":", 1)[1])
            total_tokens += token_count
        except (IndexError, ValueError):
            pass

    return {"window_seconds": window_seconds, "total_tokens": total_tokens}

Streaming Token Accumulation Protocol

The protocol for handling output token accounting under streaming has three phases, designed so that a pod crash at any phase does not result in permanent over-billing:

Phase 1 - Admission:
  reserved_tokens = input_tokens + max_tokens
  Redis: INCRBY all three window keys by reserved_tokens (Lua, atomic)
  Kafka: publish AdmissionEvent{tenant_id, reserved_tokens, admission_id}

Phase 2 - Streaming:
  For each SSE chunk:
    output_tokens += count_tokens(chunk.content)
    Redis: INCRBY quota:{tenant_id}:live:{admission_id} by chunk_tokens

Phase 3 - Completion:
  refund = reserved_tokens - input_tokens - actual_output_tokens
  Redis pipeline:
    DECRBY minute_key by refund
    DECRBY day_key by refund
    DECRBY month_key by refund
    DEL live key
  Kafka: publish UsageEvent{tenant_id, input_tokens, actual_output_tokens, cost}

Recovery (pod crash between Phase 1 and 3):
  Hourly reconciliation job scans for live keys with age > 5 minutes
  Looks up AdmissionEvent in Kafka for the admission_id
  Computes actual output tokens from the live counter at time of crash
  Issues corrective DECRBY for the unrefunded portion

Scaling and Performance

Capacity Math

At 10,000 requests per second with an average prompt of 800 input tokens and a max_tokens reservation of 1,200, each admission check writes approximately 2,000 tokens across three Redis keys. That is 30,000 INCRBY calls per second across all tenants. Redis can handle 100,000 to 300,000 simple operations per second on a single instance, so a single Redis primary handles this load easily, but we want cluster mode for high availability and for shard-level isolation.

Throughput math:
  10,000 req/s
  x 1 Lua EVAL per request (reads 4 keys, writes 3 keys)
  = 10,000 Lua EVALs/s

  Each EVAL: ~0.1ms execution on Redis (CPU-bound Lua)
  Single Redis thread: ~10,000 ops/s is 100% of one CPU

  Redis Cluster with 3 shards: tenants distributed by hash slot
  Each shard handles ~3,333 Lua EVALs/s -> ~33% CPU per shard
  Gives 3x headroom before shard saturation

Memory per tenant:
  3 string keys (minute, day, month) x 64 bytes = 192 bytes
  1 hash key (config) x ~300 bytes = 300 bytes
  1 sorted set (sliding window) x ~50 members x 80 bytes = 4,000 bytes
  Total per active tenant: ~4,500 bytes

  10,000 active tenants = 45 MB
  100,000 active tenants = 450 MB -> fits in a 1GB Redis node

Redis Cluster Sharding

Tenant IDs must hash to the same slot for all keys belonging to that tenant, or the Lua script cannot access them atomically. We use Redis hash tags to force co-location.

def make_tenant_keys(tenant_id: str) -> dict:
    """
    Use hash tags {tenant_id} so all keys for a tenant land on the same Redis slot.
    This allows the Lua script to access all keys without CROSSSLOT errors.
    """
    tag = f"{{{tenant_id}}}"
    return {
        "minute":  f"quota:{tag}:tokens:minute",
        "day":     f"quota:{tag}:tokens:day",
        "month":   f"quota:{tag}:tokens:month",
        "config":  f"quota:{tag}:config",
        "sliding": f"quota:{tag}:sliding:tokens",
    }

# Redis Cluster client configuration
import redis.asyncio as redis

async def create_redis_cluster() -> redis.RedisCluster:
    return await redis.RedisCluster.from_url(
        "redis://redis-cluster:6379",
        decode_responses=True,
        socket_connect_timeout=0.5,
        socket_timeout=0.5,
        retry_on_timeout=True,
        max_connections_per_node=50,
    )

Horizontal Scaling of the Enforcement Layer

The enforcement layer is stateless. Each pod reads from and writes to the shared Redis cluster, so adding pods adds throughput linearly up to the Redis cluster’s capacity.

Enforcement pod sizing:
  Target: 1,000 req/s per pod at <2ms p99 latency
  Each request: ~0.3ms tiktoken (CPU), ~1.2ms Redis round trip, ~0.1ms overhead
  Total: ~1.6ms mean, ~1.9ms p99

  10,000 req/s requires: 10 pods
  At peak x2 headroom: 20 pods
  Pod spec: 2 vCPU, 512MB RAM (tiktoken is the CPU consumer)

  Kubernetes HPA target: 60% CPU utilization
  Min replicas: 5 (for failure tolerance)
  Max replicas: 50 (beyond this, Redis cluster is the bottleneck)

Cost and Token Economics

Token budget management is itself a cost center. The Redis cluster, enforcement pods, and billing infrastructure consume real resources. The question is whether the control they provide justifies their operational cost.

Configuration	Monthly cost at 10K req/s	Uncontrolled overage risk	Notes
No quota enforcement	$0 infra overhead	$50,000+/month exposure	A single runaway agent loop can exhaust a month of GPU budget in hours
Post-hoc billing only	~$200/month (DB only)	$5,000-$20,000 exposure	Catches overruns too late; GPU time already consumed
Admission control (this design)	~$1,400/month	Under $200 (grace buffer only)	Redis cluster + 10 enforcement pods, PostgreSQL write path
Admission control + soft limits	~$1,600/month	Near zero	Adds alert infrastructure and webhook delivery

The $1,400/month enforcement infrastructure against a $50,000/month exposure is a 35x return on infrastructure. That calculus holds even if overruns are infrequent. One runaway production incident - an agent calling an LLM in a loop, a bug that sets max_tokens to 128,000 on every request, a tenant deliberately probing limits - recoups years of enforcement infrastructure cost.

Optimization Levers

The biggest cost lever inside the quota system itself is the refund accuracy on streaming responses. If the average request reserves 1,200 output tokens and uses 400, the over-reservation factor is 3x. This inflates effective quota consumption by 3x, requiring tenants to buy 3x more quota than they actually need, which either drives them to competitors or requires you to over-provision quota limits.

# Optimization: use model-specific output length distributions to tighten reservations
# Instead of reserving max_tokens, reserve the p95 output length for the request type

OUTPUT_LENGTH_PRIORS = {
    "classification": 15,       # 10-token label + some padding
    "summarization":  300,      # typical summary
    "qa":             150,      # typical factual answer
    "code_generation": 800,     # typical function
    "default":        512,      # conservative fallback
}

def smart_reservation(max_tokens: int, request_type: str | None) -> int:
    """
    Reserve the smaller of max_tokens and the p95 output for this request type.
    The refund mechanism handles any overshoot.
    """
    prior = OUTPUT_LENGTH_PRIORS.get(request_type or "default", 512)
    # Never reserve less than 25% of max_tokens (tail protection)
    reservation = max(max_tokens // 4, min(max_tokens, prior * 2))
    return reservation

Quality, Evaluation, and Guardrails

Token Count Accuracy

The quota system is only as good as its token counts. We measure three types of accuracy:

Admission accuracy: Does the token count used at admission (before inference) match the actual token count reported by the model after inference? For tiktoken with cl100k_base on gpt-4o models, the count is exact. For open-weight models accessed through vLLM, the HuggingFace tokenizer should produce an exact count, but there can be a 1-3 token discrepancy for inputs with special characters or Unicode that the tokenizer normalizes differently than the model.

Refund accuracy: Does actual_output_tokens reported at the end of streaming match what we counted chunk by chunk? Discrepancies arise when the model reports a different token count in its usage field than what tiktoken counts in the output text (due to special tokens not visible in the text stream).

async def verify_token_count_accuracy(
    model: str,
    prompt: str,
    actual_usage: dict,  # from API response: {"prompt_tokens": N, "completion_tokens": M}
) -> dict:
    """
    Compare our pre-inference count against the model's reported count.
    Log discrepancies above 2% for drift detection.
    """
    our_count = count_completion_tokens(model, prompt)
    model_count = actual_usage.get("prompt_tokens", 0)

    if model_count == 0:
        return {"status": "no_model_count"}

    delta = abs(our_count - model_count)
    pct_drift = (delta / max(1, model_count)) * 100

    if pct_drift > 2.0:
        # Log for drift analysis; do not block the request
        import logging
        logging.warning(
            "token_count_drift model=%s our=%d model=%d drift=%.2f%%",
            model, our_count, model_count, pct_drift,
        )

    return {
        "our_count": our_count,
        "model_count": model_count,
        "delta": delta,
        "pct_drift": pct_drift,
        "within_tolerance": pct_drift <= 2.0,
    }

Drift Detection

Counter drift accumulates when pods crash between admission and completion, when the Redis connection times out mid-refund, or when a bug in the Lua script causes incorrect increments. A weekly reconciliation job compares Redis counters against PostgreSQL aggregates and alerts on discrepancies.

async def reconcile_counters(
    redis_client: redis.Redis,
    pg_pool: asyncpg.Pool,
    tenant_id: str,
    window_date: str,  # "2026-07-29"
) -> dict:
    """
    Compare Redis day counter against PostgreSQL aggregate for drift detection.
    Run as a daily background job per tenant.
    """
    import datetime
    day_start = datetime.datetime.strptime(window_date, "%Y-%m-%d")
    unix_day = int(day_start.timestamp()) // 86400

    # Redis day counter (may have TTL-expired; 0 if expired)
    redis_key = f"quota:{tenant_id}:tokens:day:{unix_day}"
    redis_count = int(await redis_client.get(redis_key) or 0)

    # PostgreSQL authoritative sum
    pg_count = await pg_pool.fetchval(
        """
        SELECT COALESCE(SUM(input_tokens + output_tokens), 0)
        FROM usage_events
        WHERE tenant_id = $1
          AND started_at >= $2
          AND started_at < $2 + INTERVAL '1 day'
        """,
        tenant_id, day_start,
    )

    drift = abs(redis_count - pg_count)
    drift_pct = (drift / max(1, pg_count)) * 100

    return {
        "tenant_id": tenant_id,
        "window_date": window_date,
        "redis_count": redis_count,
        "pg_count": pg_count,
        "drift": drift,
        "drift_pct": drift_pct,
        "within_tolerance": drift_pct <= 0.5,
    }

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Redis primary failover	Sentinel promotes replica in 1-3s; Lua EVAL returns LOADING during transition	1-3s where admission checks fail; either all reject (fail closed) or all admit (fail open)	Default to fail-open during Redis unavailability with a circuit breaker; log every admitted request for post-hoc audit
Redis OOM - counter keys evicted	LRU eviction removes counter keys; next INCRBY restarts counter from zero	Tenant appears to have zero usage; admits over-budget requests	Set maxmemory-policy allkeys-lru and alert when memory > 80%; monitor eviction rate; use OBJECT FREQ to identify high-churn keys
Counter drift from pod crash	Reconciliation job detects Redis vs Postgres delta > 0.5%	Tenant charged less than actual (refund not applied) or more (admission count not decremented)	Reconciliation job issues corrective Redis INCRBY/DECRBY based on Kafka event log; correct delta within 1 hour
Streaming truncation at client disconnect	SSE stream ends early; completion handler triggers	If refund not applied, tenant is over-charged by max_tokens - actual_tokens	Client disconnect signal triggers refund path; test with asyncio CancelledError on the stream generator
Tokenizer version mismatch	tiktoken version upgrade changes byte-pair merges; counts drift 1-3 tokens on affected inputs	Systematic under-counting admits slightly over-budget requests	Pin tiktoken version in requirements.txt; run golden set comparison before upgrading; monitor drift metric
Lua script timeout on large sorted set	Sliding window ZRANGEBYSCORE on a tenant with millions of events	Redis blocks other commands for duration of scan; latency spike cluster-wide	Cap sorted set size with ZREMRANGEBYSCORE on every add; enforce 60-minute max TTL on all sliding window keys
Kafka consumer lag on billing path	Consumer group lag metric rises above 50,000 events	Billing aggregator falls behind; PostgreSQL usage data is stale	Scale out billing aggregator consumers; Redis is the source of truth for live quota enforcement; Kafka is only for durable billing history

Comparison of Approaches

Approach	Enforcement Latency	Accuracy	Burst Handling	Complexity	Best Fit
Token bucket (this design)	under 1ms (in-memory)	Request-level, no per-token precision	Handles bursts up to bucket capacity	Low-medium	Request rate limiting per tenant
Fixed window counter	under 2ms (Redis INCRBY)	Per-token, exact	Burst edge at window boundary (2x rate possible)	Low	Token budget enforcement per window period
Sliding window counter	2-5ms (ZRANGEBYSCORE)	Per-token, exact	Smooth, no boundary burst	Medium	Accurate rolling-window alerts and auditing
Leaky bucket	under 1ms (in-memory)	Request-level	Smoothes all bursts to constant rate	Low-medium	Strict output rate limiting for streaming
Pre-paid credits ledger	5-15ms (DB transaction)	Per-token, exact	No burst; deduct before inference	High	Marketplace billing with real money
Post-hoc billing	0ms on critical path	Per-token, exact (from model billing)	Unlimited (billed afterward)	Low	Trusted internal tenants only

For multi-tenant production use, fixed window counters in Redis with Lua atomicity are the correct choice for admission control. They run in under 2ms, are atomic, and self-expire. The sliding window belongs in the alerting path, not the admission path. The token bucket belongs at the request rate layer (requests per second), not the token budget layer (tokens per window).

Key Takeaways

• Reject before inference, not after. Post-hoc billing lets GPU cycles run for tenants who cannot pay for them. The only correct point to enforce a token budget is at admission, before any compute starts.
• Atomic check-and-decrement is non-negotiable. Read-then-write on a quota counter has a TOCTOU race that allows concurrent pods to collectively over-admit. Redis Lua scripts run atomically and eliminate this race.
• Pessimistic reservation plus refund is the streaming contract. You cannot know output tokens before generation finishes. Reserve max_tokens at admission and refund the delta on stream completion. Without the refund, quota systems over-report consumption by 2-5x.
• Input and output tokens are different prices. A system that treats them as equal will systematically misattribute cost. Use model-specific pricing tables and track them separately in the data model.
• Tokenizer must match the model. tiktoken with cl100k_base is exact for GPT-4-class models. Open-weight models need their own tokenizer. A 20 percent tokenizer mismatch means 20 percent of every request is unaccounted revenue.
• The enforcement layer must be stateless. Each enforcement pod reads from and writes to Redis. Adding pods adds throughput linearly. State in a pod means lost quota state on pod restart.
• Soft limits are operationally critical. Hard limits that fire with no warning create support escalations at midnight. An 80 percent soft-limit webhook gives tenants time to upgrade their plan or reduce their batch jobs before the hard cutoff hits.
• Reconciliation catches what atomicity misses. Lua atomicity prevents races within Redis. Pod crashes, network partitions, and streaming truncations still cause drift. A daily reconciliation job against PostgreSQL is the backstop.

Frequently Asked Questions

Q: Why use Redis for enforcement instead of a relational database?

A: The admission check happens on every single request, synchronously, before inference. At 10,000 req/s, that means 10,000 quota checks per second. A PostgreSQL UPDATE ... RETURNING with a row-level lock handles roughly 5,000 to 15,000 transactions per second on modern hardware, which gives almost no headroom for anything else. Redis processes 100,000 to 300,000 operations per second per CPU core, and a Lua script combining multiple operations takes roughly 0.1ms of Redis CPU. PostgreSQL is the authoritative store for auditing and billing; Redis is the enforcement store for latency-sensitive admission control.

Q: What happens if a tenant exhausts their minute quota but has day and month budget remaining?

A: The Lua script checks all three windows independently and rejects on the first violation. The 429 response includes retry_after_seconds computed from the time remaining in the failing window. For a minute-limit exhaustion, that is 60 seconds or less. The per-minute limit exists to prevent a single tenant from consuming the entire monthly budget in one minute-long batch job, not to throttle overall usage.

Q: How do we handle multi-turn conversations where the prompt grows with each turn?

A: Each turn is a separate admission request. The full conversation history is the prompt, so input tokens grow linearly with conversation length. A 10-turn conversation with 200 tokens per exchange has 2,000 input tokens by turn 10. Budget systems should expose a per-conversation counter to the client so it can warn users before the conversation hits the tenant’s per-minute limit mid-turn. Truncating conversation history is the application-layer workaround; the enforcement layer just counts what it receives.

Q: How do you prevent a single tenant’s sliding-window sorted set from growing unbounded?

A: Cap the TTL on the sorted set at 2x the window size (90 seconds for a 60-second window, 52 hours for a 24-hour window). On every ZADD, run ZREMRANGEBYSCORE to remove members older than the window. For the minute sliding window, a tenant doing 1,000 req/s with 1 event per request accumulates 60,000 members per minute, which is about 5 MB in Redis. Cap the set size with ZREMRANGEBYRANK to keep the last 10,000 members, which is sufficient for any reasonable burst analysis.

Q: Should the refund operation be in the same Redis pipeline as the completion event, or separate?

A: Same pipeline, wrapped in a MULTI/EXEC block. If the refund and the billing event write are not atomic, a pod crash between them leaves either an over-charged counter (refund not applied) or an unbilled event (refund applied but no billing record). In practice we use a Lua script for the refund DECRBY operations (same atomicity guarantee as the admit script) and publish the billing event to Kafka in a separate step. Kafka’s at-least-once delivery means the billing event may be written twice; the ON CONFLICT (admission_id) DO NOTHING clause in the PostgreSQL insert handles idempotency.

Q: How do you handle model pricing changes without restarting the service?

A: Store pricing in the PostgreSQL model_pricing table and cache it in Redis with a 60-second TTL. The billing aggregator reads from the cache on each batch flush and refreshes it on cache miss. The admission check does not read pricing at all - it only checks token counts against token budgets, not dollar limits. Dollar conversion happens in the billing aggregator, which runs off the critical path. This means a pricing change takes effect within 60 seconds with no deployment required.

Interview Questions

Q: Design a token budget enforcement system that handles 10,000 requests per second per tenant across three time windows (minute, day, month) with under 2ms latency. Walk through the data structures and atomicity guarantees.

Expected depth: Explain why a database transaction is too slow (row locks, round trips). Describe Redis string counters with INCRBY. Identify the TOCTOU race when check and increment are separate operations. Introduce the Lua script as the atomicity mechanism and explain why Redis evaluates Lua single-threaded. Describe the three INCRBY calls with TTL initialization. Discuss hash tags for Redis Cluster slot co-location.

Q: A streaming LLM response is cut short at 200 tokens when the client disconnects at token 50. How does this affect your quota accounting, and what do you do about it?

Expected depth: Explain pessimistic reservation: the admission check reserves max_tokens (say 200) at admission. After the client disconnects, only 50 tokens were generated. Without a refund, the tenant is over-charged by 150 tokens. Describe the stream completion handler that triggers on client disconnect (asyncio CancelledError), computes reserved - input - actual_output, and issues a DECRBY across all three window counters. Discuss the crash recovery case where the pod dies before the refund: reconciliation job reads Kafka for reserved amount, reads live counter for actual output, computes delta.

Q: You are running a Redis Cluster for token quota enforcement. Explain why a Lua script that touches multiple keys can fail with a CROSSSLOT error, and how you prevent it.

Expected depth: Redis Cluster assigns each key to a hash slot based on the key name. A Lua script can only touch keys on the same slot. If quota:tenant1:tokens:minute:X and quota:tenant1:config hash to different slots, the EVAL fails. Hash tags ({tenant1}) force Redis to compute the slot only on the bracketed portion, making all keys with {tenant1} land on the same slot. Discuss the tradeoff: all keys for a tenant are on one shard, so a hot tenant becomes a hot shard. Mitigation: add a random suffix to the tenant hash tag to spread across shards, then aggregate in the billing layer (but this breaks Lua atomicity, so it is only appropriate for monitoring, not admission).

Q: How do you ensure that the token count used for admission matches what the model actually consumed, and what do you do when they differ?

Expected depth: tiktoken is exact for OpenAI models. For open-weight models, the HuggingFace tokenizer should match but can drift by 1-3 tokens on edge cases. The model returns a usage field with prompt_tokens and completion_tokens after every request. Compare our pre-count against prompt_tokens and log any delta above 2 percent. This drift is a monitoring signal, not a blocking condition - the request already finished. The billing aggregator uses the model-reported count from the usage field, not our pre-count, so actual billing is always accurate. The pre-count just needs to be close enough for quota admission purposes.

Q: A tenant’s Redis minute counter was evicted by LRU policy during a traffic spike. The next INCRBY restarts it at zero. How do you detect and respond to this?

Expected depth: An evicted counter restarts at zero, which makes the tenant appear to have zero usage at the start of the current minute. Subsequent INCRBY calls build the counter correctly from that point, but the first few seconds of the minute are unaccounted. Detection: the reconciliation job will see a PostgreSQL aggregate higher than the Redis counter for that window. Response options: (1) set maxmemory-policy noeviction and alert on memory pressure rather than allowing eviction of quota keys; (2) use OBJECT ENCODING and OBJECT FREQ to identify high-churn keys and pin them with OBJECT PERSIST; (3) accept the under-counting and design budgets with a 5-10 percent safety margin. For a paid API, option (1) is correct - never let Redis silently drop quota counters.