Think of every LLM request as a taxi ride with no meter. A tenant sends a prompt, a GPU spins up, and the cost accumulates token by token until generation finishes. By the time you know the bill, the money is gone. For a single request that is fine. For a multi-tenant platform handling 10,000 requests per second with tenants running 128,000-token context windows at $0.06 per million output tokens, the absence of admission control converts a software bug into a $50,000 monthly overrun before the on-call engineer gets paged.

The naive approach is to track usage after the fact and cut tenants off when they exceed their quota. That approach has a fatal flaw: the GPU does not care about your accounting job. Output tokens generated by over-budget requests are real silicon cycles, real cooling watts, and real dollars billed by your cloud provider. Post-hoc accounting is a refund process dressed up as a quota system.

Correct token budget management works like a prepaid phone plan, not a credit card. Before a single GPU cycle fires, a pre-flight check verifies that the tenant has enough tokens remaining, reserves the estimated cost atomically, and either admits or rejects the request in under 2 milliseconds. Over-budget tenants hit a 429 before inference starts. GPU time goes to tenants who can pay for it.

This post designs that system end to end: the enforcement layer that runs before every request, the token counter that knows exactly how much each model will consume, the quota store that handles 10,000 concurrent atomic decrement operations in Redis, and the billing aggregator that makes the numbers match the cloud invoice.

Requirements and Constraints

Functional Requirements

• Admission control: Reject requests from tenants who have exhausted their token budget before inference starts, returning a structured 429 with retry timing.
• Real-time token counting: Count input tokens accurately per model using the correct tokenizer (cl100k_base for GPT-4-class models, a model-specific tokenizer for open-weight models) before admission.
• Output token tracking: Accumulate output tokens during streaming generation and attribute them to the originating tenant in real time.
• Budget windows: Enforce budgets at per-minute, per-day, and per-month granularities simultaneously, with configurable soft and hard limits per window.
• Cost attribution: Track input and output tokens separately because they carry different prices per model tier, and attribute costs to specific models and API keys.
• Alerts and suspension: Fire webhook alerts at configurable soft-limit thresholds (80 percent, 95 percent), suspend tenants automatically at 100 percent of the hard limit, and allow a configurable grace buffer for in-flight requests.

Non-Functional Requirements

Metric	Target	Notes
Admission control latency	under 2ms p99	Added overhead per request on the critical path
Throughput	10,000 req/s	Across all tenants simultaneously
Token count accuracy	within 2% of actual billing	tiktoken produces exact counts for OpenAI models
Counter drift	under 0.5% across budget windows	Redis atomic ops prevent split-brain; audit job detects drift
Alert delivery latency	under 5 seconds	Webhook with async queue, not blocking the request path
Quota enforcement latency	under 2ms p99	Redis Lua script evaluated atomically per request

Constraints

• No GPU waste on rejected requests. The enforcement check must complete before any model is loaded or KV cache allocated.
• Atomic decrement or reject. If two pods check a budget simultaneously and both see “budget available,” only one must win. This requires a compare-and-decrement atomic operation, not a read-then-write.
• Streaming complicates output accounting. We do not know how many output tokens a streaming request will produce until the stream ends. We must pessimistically reserve max_tokens at admission and refund the delta on completion.
• Input and output tokens price differently. A typical 70B-class model charges $0.03 per million input tokens and $0.06 per million output tokens. A budget system that treats them as the same unit will systematically undercount cost.
• Tokenizer per model. GPT-4o uses cl100k_base. Claude uses a different tokenizer. Llama 3 uses tiktoken with a SentencePiece vocabulary. Token counts are not portable across model families.

High-Level Architecture

The system breaks into four planes that operate at different latency tiers.

Enforcement Plane runs synchronously on every request path. The API Gateway calls the Budget Enforcement Layer, which counts input tokens, performs an atomic check-and-decrement in Redis, and either issues an admit decision or returns a 429. The entire round trip must complete in under 2ms.

Data Plane handles the inference itself and the real-time tracking of output tokens during streaming. The Streaming Token Counter sits between the LLM inference engine and the gateway, counting tokens as they arrive in the SSE stream and accumulating them in Redis.

Billing Plane operates asynchronously. The Billing Aggregator reads usage events from a queue, normalizes them to cost, and flushes to PostgreSQL in batches. It reconciles Redis counters against actual model billing logs to detect drift.

Alert Plane monitors thresholds and fires webhooks. It reads budget utilization from Redis on a configurable interval (typically 15 seconds) and delivers soft-limit warnings before tenants hit the hard cutoff.

The end-to-end flow for an admitted request:

1. Client sends request to API Gateway.
2. Gateway calls Budget Enforcement Layer with tenant ID, model, input text, and max_tokens.
3. Enforcement Layer tokenizes the input, adds max_tokens for a pessimistic estimate, and calls EVAL on the Redis Lua script to atomically check and decrement.
4. If admitted, the request proceeds to inference. If rejected, a 429 is returned immediately.
5. During generation, the Streaming Token Counter tallies output tokens per SSE chunk and writes them to a Redis string with INCRBY.
6. On stream end, the Enforcement Layer refunds max_tokens - actual_output_tokens back to the per-minute bucket.
7. A usage event is published to the message queue for the Billing Aggregator.
8. The Billing Aggregator batches events and writes to PostgreSQL every 5 seconds.

The Budget Enforcement Layer

The Enforcement Layer is a FastAPI service that runs synchronously on the request path. Its job is to accept or reject requests in under 2ms based on current budget state. It must handle 10,000 requests per second, which means each instance handles approximately 1,000 req/s on a 10-pod deployment.

Pre-flight Admission Control

from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse
import redis.asyncio as redis
import tiktoken
import time

app = FastAPI()
redis_client = redis.Redis(host="redis-cluster", port=6379, decode_responses=True)

# Pre-load tokenizers at startup, not per-request
TOKENIZERS = {
    "gpt-4o": tiktoken.get_encoding("cl100k_base"),
    "gpt-4o-mini": tiktoken.get_encoding("cl100k_base"),
    "gpt-4-turbo": tiktoken.get_encoding("cl100k_base"),
}

def count_tokens(model: str, text: str) -> int:
    enc = TOKENIZERS.get(model)
    if enc is None:
        # Fallback: approximate 4 chars per token for unknown models
        return max(1, len(text) // 4)
    return len(enc.encode(text))

@app.post("/v1/admit")
async def admit_request(request: Request):
    body = await request.json()
    tenant_id: str = body["tenant_id"]
    model: str = body["model"]
    prompt: str = body["prompt"]
    max_tokens: int = body.get("max_tokens", 1024)

    input_tokens = count_tokens(model, prompt)
    # Pessimistic reservation: full input + worst-case output
    reserved_tokens = input_tokens + max_tokens

    admitted, reason = await atomic_check_and_decrement(
        tenant_id=tenant_id,
        model=model,
        input_tokens=input_tokens,
        reserved_tokens=reserved_tokens,
    )

    if not admitted:
        raise HTTPException(
            status_code=429,
            detail={
                "error": "token_budget_exceeded",
                "reason": reason,
                "tenant_id": tenant_id,
                "retry_after_seconds": await seconds_until_window_reset(tenant_id),
            },
        )

    return {
        "admitted": True,
        "input_tokens": input_tokens,
        "reserved_tokens": reserved_tokens,
        "admission_id": f"{tenant_id}:{int(time.time() * 1000)}",
    }

Redis Lua Script for Atomic Check-and-Decrement

The core of admission control is a Lua script that Redis evaluates as a single atomic transaction. No other command from any client runs between the reads and the decrements.

-- atomic_admit.lua
-- KEYS[1] = per-minute counter key  e.g. quota:tenant123:tokens:minute:2026072914
-- KEYS[2] = per-day counter key      e.g. quota:tenant123:tokens:day:20260729
-- KEYS[3] = per-month counter key    e.g. quota:tenant123:tokens:month:202607
-- KEYS[4] = tenant config hash key   e.g. quota:tenant123:config
-- ARGV[1] = reserved_tokens (input + max_output)
-- ARGV[2] = current unix timestamp (for TTL initialization)
-- Returns: {"ok", input_tokens_used} or {"err", reason}

local reserved  = tonumber(ARGV[1])
local now       = tonumber(ARGV[2])

-- Load tenant config
local minute_limit = tonumber(redis.call('HGET', KEYS[4], 'minute_limit') or 0)
local day_limit    = tonumber(redis.call('HGET', KEYS[4], 'day_limit')    or 0)
local month_limit  = tonumber(redis.call('HGET', KEYS[4], 'month_limit')  or 0)
local suspended    = redis.call('HGET', KEYS[4], 'suspended')

if suspended == '1' then
    return {'err', 'account_suspended'}
end

-- Read current counters (GETEX would be cleaner but KEYS[n] pattern needs GET)
local used_minute = tonumber(redis.call('GET', KEYS[1]) or 0)
local used_day    = tonumber(redis.call('GET', KEYS[2]) or 0)
local used_month  = tonumber(redis.call('GET', KEYS[3]) or 0)

-- Check all three windows before touching any counter
if minute_limit > 0 and (used_minute + reserved) > minute_limit then
    return {'err', 'minute_limit_exceeded'}
end
if day_limit > 0 and (used_day + reserved) > day_limit then
    return {'err', 'day_limit_exceeded'}
end
if month_limit > 0 and (used_month + reserved) > month_limit then
    return {'err', 'month_limit_exceeded'}
end

-- All checks passed; increment all three windows atomically
local new_minute = redis.call('INCRBY', KEYS[1], reserved)
local new_day    = redis.call('INCRBY', KEYS[2], reserved)
local new_month  = redis.call('INCRBY', KEYS[3], reserved)

-- Set TTL only if the key was just created (INCRBY on a missing key starts at 0)
-- 90 seconds for minute window, 26 hours for day, 35 days for month
if new_minute == reserved then redis.call('EXPIRE', KEYS[1], 90) end
if new_day    == reserved then redis.call('EXPIRE', KEYS[2], 93600) end
if new_month  == reserved then redis.call('EXPIRE', KEYS[3], 3024000) end

return {'ok', tostring(new_minute), tostring(new_day), tostring(new_month)}

# Calling the Lua script from Python
LUA_SCRIPT = open("atomic_admit.lua").read()
_admit_script = None

async def get_admit_script():
    global _admit_script
    if _admit_script is None:
        _admit_script = redis_client.register_script(LUA_SCRIPT)
    return _admit_script

async def atomic_check_and_decrement(
    tenant_id: str,
    model: str,
    input_tokens: int,
    reserved_tokens: int,
) -> tuple[bool, str]:
    now = int(time.time())
    minute_key = f"quota:{tenant_id}:tokens:minute:{now // 60}"
    day_key    = f"quota:{tenant_id}:tokens:day:{now // 86400}"
    month_key  = f"quota:{tenant_id}:tokens:month:{now // 2592000}"
    config_key = f"quota:{tenant_id}:config"

    script = await get_admit_script()
    result = await script(
        keys=[minute_key, day_key, month_key, config_key],
        args=[str(reserved_tokens), str(now)],
    )
    status = result[0]
    if status == "ok":
        return True, ""
    return False, result[1]