A restaurant kitchen that only ever cooked one dish would be easy to run: one prep station, one burner schedule, one ticket rail. A kitchen that has to fire a seared steak, a delicate souffle, and a flash-boiled noodle dish off the same pass, on the same clock, with the same head chef, is a different problem entirely. Each dish has its own prep time, its own equipment, and its own tolerance for sitting under a heat lamp. That is the problem a multi-modal inference server has to solve: text prompts, images, and audio clips arrive on the same endpoint, need wildly different amounts of compute to turn into tokens, and all have to land in the same decoder so the model can reason across all three.

The scale numbers make the tension concrete. A production multi-modal endpoint serving a consumer assistant sees on the order of 250 requests per second blended across modalities. A single image, resized to 336x336 and patchified the way LLaVA and Qwen-VL do it, expands into 576 visual tokens before the decoder ever sees a word of the prompt. A 20-second audio clip, after a Whisper-style encoder compresses it, lands around 500 audio tokens. A plain text turn averages 300 tokens. Encoding an image on a vision tower takes 40-90ms of dense compute on a ViT; encoding audio takes longer because the encoder runs over the whole waveform sequentially. None of that overlaps for free with the autoregressive decode loop that has to hit a time-to-first-token (TTFT) budget under a second and keep generating at a steady time-per-output-token (TPOT).

The naive approach is to treat every request the same way: run the request through whatever encoder it needs, concatenate the output tokens with the text prompt, and hand the whole sequence to the decoder in the order it arrived. That falls apart in three places. First, image and audio encoding are throughput-bound batch operations that want to run many inputs at once on a GPU, while decode is latency-bound and wants small, fast steps; queuing them behind the same worker means a burst of images stalls every in-flight text-only decode. Second, the decoder’s KV cache has to hold 500-1000+ extra tokens per multi-modal request compared to a text-only one, so treating every request as equally expensive to admit blows the memory budget. Third, caching only helps if you cache the right thing at the right layer: caching a whole response is useless when the image changes every request, but caching the encoded representation of a repeated image (a product photo shown to 10,000 users) is enormously valuable.

We need to solve for heterogeneous compute scheduling across encoders and decoder, token-budget-aware admission control so multi-modal requests do not silently starve the KV cache, and modality-specific caching so repeated media never gets re-encoded. Get those three right and the rest of the system - routing, batching, streaming - falls into place around them.

Requirements and Constraints

Functional Requirements

• Accept requests containing any combination of a text prompt, one or more images, and one audio clip, and produce a single coherent streamed text response.
• Detect the modality mix per request and route each non-text input to the correct encoder (vision tower for images, audio encoder for speech/sound) before any decoding starts.
• Fuse encoder outputs and text tokens into a single ordered sequence the decoder can attend over, preserving positional relationships between the media and the surrounding text.
• Stream tokens back to the client over SSE as soon as the first token is available, regardless of which modalities were present.
• Cache encoder outputs for repeated media (same image bytes, same audio hash) so identical inputs never pay encoding cost twice.
• Support at least two model sizes (a fast 8B-class decoder and a stronger 70B-class decoder) with routing based on request complexity or explicit client hint.

Non-Functional Requirements

• Latency: p50 TTFT under 400ms for text-only requests, p99 TTFT under 900ms when an image or audio encode is on the critical path; TPOT under 35ms per token at p50.
• Throughput: sustain 250 requests/second blended traffic, with peak bursts to 600 requests/second for 2-3 minutes during traffic spikes.
• Quality: cross-modal grounding accuracy (does the generated text correctly reference the image/audio content) above 92% on a held-out eval set; hallucinated object references under 3%.
• Cost: median cost per 1,000 requests under $4.50 blended across modalities; image and audio encoding must not add more than 25% to the per-request GPU-second cost of an equivalent text-only request.
• Capacity: KV cache budget per node must reserve headroom for the 99th-percentile multi-modal token count (roughly 1,100 fused tokens of context before generation starts), not just the text-only average.

Constraints

• Decoder: a single family of instruction-tuned multi-modal-capable LLMs (8B and 70B variants) served with vLLM-style continuous batching and PagedAttention.
• Vision tower: a frozen ViT-L/14-class encoder producing 576 tokens per 336x336 image tile, projected into the decoder’s embedding space via a learned adapter.
• Audio encoder: a Whisper-large-class encoder producing a compressed token stream at roughly 25 tokens per second of audio.
• Hardware: 8x A100 80GB GPU nodes, with dedicated node pools for encoders versus decoder rather than colocating everything on one GPU.
• Out of scope: image or audio generation (this server only consumes media, it does not produce it), video input, and real-time bidirectional voice (this is turn-based, not a live voice call).

High-Level Architecture

The system has six major components: an ingress gateway that accepts and validates multi-modal requests, a modality router that inspects each request and dispatches non-text payloads to the right encoder pool, a vision encoder pool and an audio encoder pool that each turn raw media into token sequences on dedicated GPUs, a token fuser that merges encoder outputs with text tokens and manages a prefix cache for repeated media, and a decoder serving pool that runs continuous batching and PagedAttention over the fused sequences and streams tokens back to the client.

A request lands at the ingress gateway carrying a text prompt and, optionally, one or more image blobs and an audio blob. The gateway validates payload sizes, strips EXIF and other metadata from images, and forwards the request to the modality router with a manifest describing what’s present. The router does not do any encoding itself - its only job is to fan out the image and audio payloads to their respective encoder pools while the text prompt waits in the token fuser. Each encoder pool batches inputs across requests (many images from different users can share one forward pass through the ViT), checks the embedding cache first, and returns a token sequence plus positional metadata back to the token fuser.

The token fuser is the seam between “encoding” and “decoding.” It assembles the final input sequence - text tokens, visual tokens, audio tokens, in the order the client specified - and hands the completed sequence to the decoder pool for prefill and generation. The decoder pool treats a fused multi-modal sequence exactly like a long text sequence for scheduling purposes: it goes through the same continuous batching admission logic, occupies PagedAttention blocks in the KV cache, and streams tokens back through the gateway over SSE.

The Modality Router

The modality router’s job is to look at a request’s manifest and decide, in under a millisecond, which encoder pools need to see this request and in what order their outputs must be stitched back together.

The non-obvious part is that routing is not just “if image, call vision encoder.” Some requests need map semantics (one image, one text block) and some need interleaved semantics (multiple images referenced inline within the text, like “compare the first photo to the second one”). The router has to preserve a stable ordering token - a placeholder ID for each media item - so the fuser can reinsert encoder outputs at exactly the right position in the final sequence, even though encoding happens asynchronously and out of order relative to other requests.

# modality_router.py - dispatches media placeholders to encoder pools
# and tracks per-request completion before the fuser can proceed
import asyncio
import hashlib
from dataclasses import dataclass, field
from enum import Enum

class MediaKind(str, Enum):
    IMAGE = "image"
    AUDIO = "audio"

@dataclass
class MediaRef:
    kind: MediaKind
    placeholder_id: str
    content_hash: str
    position: int  # token index in the original prompt where this slots in

@dataclass
class RoutedRequest:
    request_id: str
    text_prompt: str
    media_refs: list[MediaRef] = field(default_factory=list)
    pending: set[str] = field(default_factory=set)

def build_content_hash(raw_bytes: bytes) -> str:
    return hashlib.sha256(raw_bytes).hexdigest()[:24]

async def route_request(req_id: str, text: str, images: list[bytes], audio: bytes | None,
                         vision_queue: asyncio.Queue, audio_queue: asyncio.Queue) -> RoutedRequest:
    routed = RoutedRequest(request_id=req_id, text_prompt=text)
    for idx, img_bytes in enumerate(images):
        ref = MediaRef(
            kind=MediaKind.IMAGE,
            placeholder_id=f"{req_id}-img-{idx}",
            content_hash=build_content_hash(img_bytes),
            position=text.find(f"<image-{idx}>"),
        )
        routed.media_refs.append(ref)
        routed.pending.add(ref.placeholder_id)
        await vision_queue.put((ref, img_bytes))
    if audio is not None:
        ref = MediaRef(
            kind=MediaKind.AUDIO,
            placeholder_id=f"{req_id}-audio-0",
            content_hash=build_content_hash(audio),
            position=text.find("<audio-0>"),
        )
        routed.media_refs.append(ref)
        routed.pending.add(ref.placeholder_id)
        await audio_queue.put((ref, audio))
    return routed

The Vision Encoder Pool

The vision encoder pool’s job is to turn raw image bytes into a fixed-length sequence of embedding vectors the decoder can treat as tokens, as cheaply as possible per image.

A smart engineer’s first assumption is usually that image encoding is cheap compared to decoding a 70B model, so it can just run inline on whichever GPU handles the request. That is wrong once you account for batch efficiency: a ViT-L/14 forward pass is compute-bound and benefits enormously from batching 16-64 images together, the same way a bakery is more efficient baking a full tray than one loaf at a time. Running it inline, one image per call, on the decoder’s GPU wastes the tensor cores and steals memory bandwidth from the exact node that’s trying to hit a decode latency SLA.

The pool instead runs as its own replica set on dedicated GPUs, with a short (10-15ms) micro-batching window that groups images arriving close together into one forward pass, the same pattern continuous batching uses on the decoder side but tuned for a much shorter, throughput-bound operation.

# vision_encoder_pool.py - micro-batches images before a single ViT forward pass
import asyncio
import torch
from torchvision import transforms

PATCH_TOKENS_PER_IMAGE = 576
MAX_BATCH = 48
BATCH_WINDOW_MS = 12

preprocess = transforms.Compose([
    transforms.Resize((336, 336)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.481, 0.458, 0.408], std=[0.269, 0.261, 0.276]),
])

class VisionEncoderPool:
    def __init__(self, vit_model, adapter, cache_client, device="cuda:0"):
        self.vit_model = vit_model.eval().to(device)
        self.adapter = adapter.eval().to(device)
        self.device = device
        self.cache = cache_client
        self.inbox: asyncio.Queue = asyncio.Queue()

    async def run(self):
        while True:
            batch = [await self.inbox.get()]
            try:
                while len(batch) < MAX_BATCH:
                    item = await asyncio.wait_for(self.inbox.get(), timeout=BATCH_WINDOW_MS / 1000)
                    batch.append(item)
            except asyncio.TimeoutError:
                pass
            await self._encode_batch(batch)

    async def _encode_batch(self, batch):
        uncached, cached_refs = [], []
        for ref, img_bytes, future in batch:
            cached = await self.cache.get(f"vis:{ref.content_hash}")
            if cached is not None:
                cached_refs.append((ref, cached, future))
            else:
                uncached.append((ref, img_bytes, future))

        for ref, tokens, future in cached_refs:
            future.set_result(tokens)

        if not uncached:
            return

        with torch.no_grad():
            tensors = torch.stack([preprocess(_decode_image(b)) for _, b, _ in uncached]).to(self.device)
            hidden = self.vit_model(tensors)              # [batch, 576, vit_dim]
            projected = self.adapter(hidden)               # [batch, 576, decoder_dim]

        for (ref, _, future), emb in zip(uncached, projected):
            tokens = emb.cpu()
            await self.cache.set(f"vis:{ref.content_hash}", tokens, ttl_seconds=86400)
            future.set_result(tokens)

def _decode_image(raw_bytes: bytes):
    from PIL import Image
    import io
    return Image.open(io.BytesIO(raw_bytes)).convert("RGB")

Skip the cache and every repeated product photo, avatar, or screenshot re-runs the full ViT forward pass, which at 250 req/s with even a 20% repeat rate wastes a meaningful slice of the vision pool’s GPU-seconds on work it already did.

The Audio Encoder Pool

The audio encoder pool’s job is to convert a variable-length waveform into a compact token sequence proportional to audio duration, without forcing the caller to wait for the entire clip to transcribe before decoding can start.

The instinct to run a full ASR pass (transcribe to text, then feed text to the LLM) throws away information the model could otherwise use - tone, emphasis, non-speech sounds - and adds a second model’s latency in serial with the first. Instead, the pool runs a Whisper-style encoder only (not the full transcribe-and-decode ASR pipeline) and feeds its continuous hidden states directly into the fuser as audio tokens, the way Qwen2-Audio and similar audio-LLMs do it. This keeps richer signal than a transcript and avoids a second full inference pass.

# audio_encoder_pool.py - streams a Whisper-style encoder over fixed windows
# and emits ~25 tokens/sec of audio without waiting for full-clip transcription
import asyncio
import torch

TOKENS_PER_SECOND = 25
WINDOW_SECONDS = 5
SAMPLE_RATE = 16000

class AudioEncoderPool:
    def __init__(self, whisper_encoder, adapter, cache_client, device="cuda:1"):
        self.encoder = whisper_encoder.eval().to(device)
        self.adapter = adapter.eval().to(device)
        self.device = device
        self.cache = cache_client

    async def encode(self, ref, waveform: torch.Tensor) -> torch.Tensor:
        cache_key = f"aud:{ref.content_hash}"
        cached = await self.cache.get(cache_key)
        if cached is not None:
            return cached

        num_samples = waveform.shape[-1]
        window_samples = WINDOW_SECONDS * SAMPLE_RATE
        chunks = [
            waveform[i:i + window_samples]
            for i in range(0, num_samples, window_samples)
        ]

        all_tokens = []
        with torch.no_grad():
            for chunk in chunks:
                padded = torch.nn.functional.pad(chunk, (0, window_samples - chunk.shape[-1]))
                hidden = self.encoder(padded.unsqueeze(0).to(self.device))
                projected = self.adapter(hidden)
                all_tokens.append(projected.squeeze(0))

        tokens = torch.cat(all_tokens, dim=0)
        expected = int((num_samples / SAMPLE_RATE) * TOKENS_PER_SECOND)
        tokens = tokens[:max(expected, 1)]
        await self.cache.set(cache_key, tokens, ttl_seconds=3600)
        return tokens

If a 60-second clip is fed through with no windowing, the encoder has to hold the entire waveform’s activations in memory at once and the caller sees zero progress until the whole thing finishes; windowing at 5 seconds means the fuser can start receiving audio tokens for the first window while later windows are still encoding, which shrinks TTFT for audio-heavy requests.

The Token Fuser and Prefix Cache

The token fuser’s job is to assemble one ordered token sequence from text, visual, and audio pieces, and to keep that assembly on the decoder’s KV cache prefix path so repeated context is never recomputed.

The subtlety most engineers miss here is treating fusion as a formatting step. It isn’t - fusion determines where in the KV cache prefill each media block lands, and the decoder’s attention pattern is sensitive to that ordering. A system prompt plus a fixed reference image (a product catalog thumbnail shown on every request in a shopping assistant) is exactly the kind of prefix that should hit the decoder’s PagedAttention prefix cache and skip re-running attention over those 576 tokens on every single request.

# token_fuser.py - assembles fused sequences and tags cacheable prefixes
from dataclasses import dataclass

@dataclass
class FusedSegment:
    kind: str          # "text" | "vision" | "audio"
    tokens: "torch.Tensor | list[int]"
    cacheable: bool     # True if this segment is stable across requests (e.g. system prompt + fixed image)

def fuse(routed_request, encoder_results: dict[str, "torch.Tensor"], tokenizer) -> list[FusedSegment]:
    text = routed_request.text_prompt
    segments: list[FusedSegment] = []
    cursor = 0
    ordered_refs = sorted(routed_request.media_refs, key=lambda r: r.position)

    for ref in ordered_refs:
        if ref.position > cursor:
            chunk = text[cursor:ref.position]
            segments.append(FusedSegment("text", tokenizer.encode(chunk), cacheable=_is_system_prefix(chunk)))
        media_tokens = encoder_results[ref.placeholder_id]
        segments.append(FusedSegment(ref.kind.value, media_tokens, cacheable=ref.kind.value == "vision" and _is_pinned_reference(ref)))
        cursor = ref.position + len(f"<{ref.kind.value}-0>")

    if cursor < len(text):
        segments.append(FusedSegment("text", tokenizer.encode(text[cursor:]), cacheable=False))
    return segments

def _is_system_prefix(chunk: str) -> bool:
    return chunk.strip().startswith("SYSTEM:")

def _is_pinned_reference(ref) -> bool:
    return ref.content_hash in PINNED_REFERENCE_HASHES

PINNED_REFERENCE_HASHES: set[str] = set()

The Continuous Batching Decoder Pool

The decoder pool’s job is to run prefill and autoregressive decode over fused multi-modal sequences at the throughput and latency the SLA demands, without letting expensive multi-modal prefills starve cheap text-only ones.

The assumption that trips people up is that continuous batching treats all requests as equally expensive per token. A fused multi-modal sequence’s prefill is 3-5x more expensive than a text-only one at the same token count because the KV cache blocks for visual and audio tokens still have to go through the same attention computation - the tokens are indistinguishable from text tokens to the attention mechanism itself. Admission control has to account for this or a burst of image-heavy requests will blow the per-node KV cache budget even though the request count looks fine.

# admission_control.py - blocks admission when fused KV footprint would
# exceed the node's PagedAttention block budget
BLOCK_SIZE_TOKENS = 16
TOTAL_BLOCKS_PER_NODE = 8_000        # ~80GB A100, 70B model, FP16 KV cache
RESERVED_BLOCKS_FOR_DECODE = 1_200   # headroom for in-flight decode steps

class AdmissionController:
    def __init__(self):
        self.free_blocks = TOTAL_BLOCKS_PER_NODE

    def can_admit(self, fused_token_count: int) -> bool:
        needed_blocks = -(-fused_token_count // BLOCK_SIZE_TOKENS)  # ceil div
        return (self.free_blocks - needed_blocks) >= RESERVED_BLOCKS_FOR_DECODE

    def admit(self, fused_token_count: int) -> int:
        needed_blocks = -(-fused_token_count // BLOCK_SIZE_TOKENS)
        self.free_blocks -= needed_blocks
        return needed_blocks

    def release(self, blocks_held: int):
        self.free_blocks += blocks_held

Without this check, the scheduler admits requests purely by count or by a flat token estimate, and a wave of three-image requests (1,728 visual tokens each, before any text) can consume the entire block pool, forcing every other in-flight decode to pause mid-generation while the scheduler evicts or waits, which shows up to users as a multi-second stall with no warning.

Data Model

The system tracks three kinds of records: request-level metadata for billing and tracing, cached encoder outputs keyed by content hash, and the fused-sequence cache entries that back the decoder’s prefix cache.

-- core relational schema: requests, media cache index, and quota tracking
CREATE TABLE inference_requests (
    request_id      UUID PRIMARY KEY,
    tenant_id        UUID NOT NULL,
    modality_mask    SMALLINT NOT NULL,       -- bitmask: 1=text, 2=image, 4=audio
    text_tokens      INTEGER NOT NULL DEFAULT 0,
    vision_tokens    INTEGER NOT NULL DEFAULT 0,
    audio_tokens     INTEGER NOT NULL DEFAULT 0,
    ttft_ms          INTEGER,
    total_latency_ms INTEGER,
    model_variant    TEXT NOT NULL,           -- '8b' | '70b'
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT chk_modality_mask CHECK (modality_mask BETWEEN 1 AND 7)
);
CREATE INDEX idx_requests_tenant_time ON inference_requests (tenant_id, created_at DESC);

CREATE TABLE media_embedding_cache (
    content_hash    CHAR(24) PRIMARY KEY,
    media_kind      TEXT NOT NULL CHECK (media_kind IN ('image', 'audio')),
    token_count     INTEGER NOT NULL,
    storage_uri      TEXT NOT NULL,           -- object store path to the serialized tensor
    hit_count       BIGINT NOT NULL DEFAULT 0,
    last_hit_at     TIMESTAMPTZ,
    expires_at      TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_media_cache_expiry ON media_embedding_cache (expires_at);

CREATE TABLE tenant_quota (
    tenant_id           UUID PRIMARY KEY,
    tokens_per_minute   INTEGER NOT NULL,
    concurrent_requests INTEGER NOT NULL,
    tier                TEXT NOT NULL DEFAULT 'standard'
);

The embedding cache is partitioned by content_hash rather than tenant, because the whole point is that identical media from different tenants (a shared stock photo, a common notification sound) should still hit the cache. Vector-style records only show up here indirectly - the cached “vectors” are per-token embedding sequences (576 rows for an image, ~25/sec for audio), stored as serialized tensors in object storage with the SQL row acting as an index, not as a similarity-search index. There is no approximate nearest-neighbor lookup in this system; the lookup key is an exact content hash, so no HNSW-style index is needed - a plain B-tree primary key index on content_hash is enough.

Key Algorithms and Protocols

Three algorithms carry the system’s weight: continuous batching admission with modality-aware cost, PagedAttention block allocation across mixed-length prefixes, and prefix-cache matching for fused sequences that share a system prompt or pinned image.

Continuous batching with modality-aware cost. Standard continuous batching admits requests into the running batch as soon as KV cache space frees up, estimating cost by token count. Here, the estimate has to weight visual and audio tokens by their actual attention cost rather than treating a token as a token, because a 576-token image block occupies the same KV cache space as 576 text tokens but arrived as one atomic unit that cannot be partially admitted.

# continuous_batch_scheduler.py - selects the next requests to admit into
# the running decode batch, respecting per-node block budget
from dataclasses import dataclass

@dataclass
class Candidate:
    request_id: str
    fused_token_count: int
    priority: int  # lower = higher priority (e.g. interactive vs batch tier)

def select_next_batch(candidates: list[Candidate], controller, max_batch_size: int) -> list[Candidate]:
    ordered = sorted(candidates, key=lambda c: (c.priority, c.fused_token_count))
    admitted = []
    for cand in ordered:
        if len(admitted) >= max_batch_size:
            break
        if controller.can_admit(cand.fused_token_count):
            controller.admit(cand.fused_token_count)
            admitted.append(cand)
    return admitted

Sorting by (priority, fused_token_count) means cheap requests fill any remaining headroom after high-priority ones are admitted, which keeps GPU utilization high without letting one enormous multi-image request monopolize an entire batching round. Complexity is O(n log n) for the sort plus O(n) for the admission scan, dominated by candidate list size, which stays in the low hundreds per scheduling tick even at 600 req/s bursts.

Prefix cache matching for fused sequences. The decoder’s prefix cache indexes KV blocks by a hash of the token sequence up to that point. For fused sequences, the hash has to be computed over the concatenated token IDs of text and media segments in order, so two requests sharing the same system prompt and pinned reference image - but different user questions - still match on the shared prefix and only pay prefill cost for the divergent suffix.

# prefix_cache.py - hashes fused segments incrementally so shared
# system-prompt-plus-image prefixes reuse KV blocks across requests
import hashlib

def prefix_hashes(segments: list) -> list[str]:
    running = hashlib.sha256()
    hashes = []
    for seg in segments:
        token_bytes = _tokens_to_bytes(seg.tokens)
        running.update(token_bytes)
        hashes.append(running.hexdigest())
    return hashes

def _tokens_to_bytes(tokens) -> bytes:
    if hasattr(tokens, "numpy"):
        return tokens.numpy().tobytes()
    return bytes(tokens)

def find_longest_cached_prefix(hashes: list[str], kv_block_index: dict) -> int:
    matched = 0
    for h in hashes:
        if h in kv_block_index:
            matched += 1
        else:
            break
    return matched

Edge case: if the pinned image is swapped (a catalog refresh) but the content hash logic upstream is not invalidated in lockstep, the prefix hash silently diverges on the first token after the stale image block and the cache simply misses cleanly - it cannot serve stale visual tokens, because the hash chain breaks at the exact point the content changed.

Scaling and Performance

The decoder pool scales horizontally by adding replicas behind the continuous batching scheduler, but a single replica’s KV cache capacity does not scale - it is bounded by that node’s GPU memory once model weights are loaded. The vision and audio encoder pools scale independently and more cheaply, since they hold no persistent per-request state beyond the current micro-batch.

Given:
  - 250 req/s blended, 40% image-included, 15% audio-included
  - 70B decoder in FP16 = 140 GB weights -> 2x A100 80GB with tensor parallelism
  - Avg fused prompt: 300 text + 0.4*576 image + 0.15*500 audio ~= 611 tokens
  - Avg output: 250 tokens

KV cache per seq: 80 layers * 8 kv heads * 128 dim * 2 * 2B * (611 + 250) tok = ~0.28 GB
Per node (2x A100, 160GB - 140GB weights = 20GB usable): 20 / 0.28 = ~71 concurrent seqs
Decode throughput per node: 71 seqs * ~35 tok/s = ~2,485 tok/s
Total output tokens needed: 250 req/s * 250 tok = 62,500 tok/s
Decoder nodes needed: 62,500 / 2,485 = ~26 nodes (52x A100)

Vision encoder: 250 * 0.4 = 100 img/s, ViT batch of 48 @ 35ms -> ~1,371 img/s per GPU
Vision GPUs needed: 100 / 1,371 = ~1 GPU (round up to 2 for headroom)

Audio encoder: 250 * 0.15 = 37.5 clips/s, avg 20s clip, encoder real-time factor ~8x
Audio GPUs needed: (37.5 * 20) / (8 * 60) = ~1.6 GPUs (round up to 2)

The decoder dominates the GPU footprint by an order of magnitude, which is why the encoder pools get a small, separate node pool rather than a fraction of the decoder’s nodes - colocating them would mean scaling 26 expensive tensor-parallel decoder nodes just to add encoder headroom for a traffic pattern that only needs 2-4 GPUs worth of encoding capacity.

Caching strategy has three tiers: the media embedding cache (read-heavy, TTL-based, keyed by content hash) absorbs repeated images and audio; the KV prefix cache (write-once-per-unique-prefix, read on every request sharing that prefix) absorbs shared system prompts and pinned references; and a short-lived response cache is deliberately not used, since multi-modal responses are rarely byte-identical across users even when the media matches.

Cost and Token Economics

Cost has three drivers: decoder GPU-hours (dominant, as the capacity math above shows), encoder GPU-hours (small but non-zero), and cache storage for embeddings (cheap, but grows with unique media volume).

Configuration	Cost per 1,000 requests	TTFT p99	Notes
70B decoder, no embedding cache	$6.10	1,050ms	Every repeated image/audio re-encoded every time
70B decoder, with embedding cache (30% hit rate)	$4.35	780ms	Typical production hit rate for consumer traffic
8B decoder for simple queries, 70B for complex, with cache	$2.90	620ms	Router downgrades text-only or short queries to 8B
API provider (pay-per-token multi-modal API)	$8.70	900ms	No infra to run, but image tokens billed at a premium multiplier

The single highest-leverage optimization is the embedding cache: at a measured 30% hit rate on production-like traffic (repeated avatars, product photos, and notification sounds), it cuts blended cost per 1,000 requests by roughly 29% and cuts p99 TTFT by 26%, because a cache hit skips both the encoder GPU-seconds and the queueing delay behind the encoder’s micro-batch window entirely.

Quality, Evaluation, and Guardrails

AI systems fail softly - a multi-modal server does not crash when it misreads an image, it confidently describes the wrong object. Correctness has to be measured, not assumed.

Offline evals run a golden set of image-question and audio-question pairs with known-correct answers through the pipeline nightly, scoring cross-modal grounding with an LLM-as-judge pass that checks whether the generated answer’s claims are actually supported by the referenced media. Online, the system tracks thumbs-up/down rate per modality mix, regeneration rate (a strong proxy for “the first answer ignored the image”), and a groundedness score sampled on a rolling basis. A deploy is gated on grounding accuracy staying above 90% and hallucinated-object rate staying under 4%; either metric breaching threshold on the canary triggers an automatic rollback.

# grounding_guardrail.py - flags generated answers that reference objects
# or claims not present in the vision encoder's detected content
def check_grounding(generated_text: str, detected_labels: set[str], confidence_threshold: float = 0.6) -> dict:
    claimed_objects = extract_object_mentions(generated_text)  # NER-style extraction
    unsupported = [obj for obj in claimed_objects if obj.lower() not in detected_labels]

    grounded_ratio = 1.0 if not claimed_objects else 1 - (len(unsupported) / len(claimed_objects))
    return {
        "grounded_ratio": grounded_ratio,
        "unsupported_claims": unsupported,
        "passes_guardrail": grounded_ratio >= confidence_threshold,
    }

def extract_object_mentions(text: str) -> list[str]:
    # placeholder for a real NER pass; production uses a small classifier
    # tuned on the domain's object vocabulary
    common_nouns = ["dog", "cat", "car", "person", "building", "sign", "food"]
    return [w for w in common_nouns if w in text.lower()]

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
GPU OOM mid-decode from misestimated multi-modal token cost	CUDA OOM exception on the decoder process, spike in admission controller rejections	In-flight sequences on that node fail, client sees a dropped stream	Preempt lowest-priority sequences, requeue on another replica, tighten RESERVED_BLOCKS_FOR_DECODE
Vision encoder pool cold start after autoscale-up	Health check latency spike, first-batch latency 5-10x normal	Image requests queue behind a slow warm-up batch	Pre-warm replicas with a dummy batch on startup before registering with the load balancer
Audio codec mismatch producing garbage waveform samples	Encoder output NaN check fails, or duration mismatch between claimed and decoded audio	Model receives corrupted audio tokens, generates unrelated or garbled text	Reject at ingress with a validation error rather than passing corrupted tensors downstream
Embedding cache poisoned by a hash collision or stale entry after content update	Hit-rate monitor flags an unexpected spike in cache hits, correctness eval flags mismatched grounding	Wrong cached embedding served for new content, model reasons about the wrong image	Content-address the cache by full hash (not truncated), invalidate on any upstream content version bump
Upstream 429 from a hosted vision or audio model dependency	Encoder pool sees repeated 429 responses, request latency climbs	Requests needing that encoder queue or time out	Circuit-break to a smaller fallback encoder, degrade gracefully to text-only response with a note that media could not be processed
Truncated SSE stream mid-generation from a client disconnect or proxy timeout	Gateway detects closed connection, decoder continues generating into a dead stream	Wasted GPU-seconds on unread tokens	Cancel the decode on disconnect detection, free the KV cache blocks immediately rather than waiting for natural completion

Comparison of Approaches

Approach	Latency	Cost	Complexity	Best fit scenario
Separate models per modality (ASR then text-LLM, caption-then-text-LLM)	Higher (serial passes)	Lower per-model, higher total	Low - each model is simple to operate	Early-stage products validating demand before investing in a fused architecture
Native multi-modal decoder with fused tokens (this design)	Lower (parallel encode, single decode pass)	Higher decoder cost, offset by caching	High - requires coordinated scheduling across pools	Production systems with sustained mixed-modality traffic and grounding quality requirements
Hosted multi-modal API (pay-per-token)	Comparable to self-hosted at low volume	Higher at scale, no infra ops cost	Lowest - no serving infra to run	Low-to-medium volume products, or teams without ML infra capacity
Colocated encoders and decoder on one GPU pool	Highest under load (resource contention)	Lower node count, higher tail latency cost	Medium - simpler topology, harder to tune	Prototypes and low-traffic internal tools where tail latency doesn’t matter

For sustained production traffic above roughly 50 req/s with a real mix of modalities, the native fused decoder with separated encoder pools is worth the added scheduling complexity - the cost and latency wins from independent scaling and the embedding cache compound quickly, while the hosted-API and colocated options both hit a wall (billing cost or tail latency, respectively) well before that volume.

Key Takeaways

• Heterogeneous scheduling is the core problem - vision and audio encoding are throughput-bound batch operations, decode is latency-bound, and conflating them on shared infrastructure degrades both.
• Token-cost-aware admission control prevents multi-modal requests from silently exhausting the KV cache budget that flat request-count-based admission would miss.
• Content-hash-keyed embedding caching is the single highest-leverage cost and latency optimization, cutting both by roughly a quarter at realistic hit rates.
• Prefix caching for fused sequences requires chaining hashes across text and media segments so shared system prompts and pinned images reuse KV blocks safely.
• Separating encoder and decoder GPU pools lets each scale against its own bottleneck instead of both scaling against whichever one is currently under more load.
• Grounding evals must be sliced by modality mix, since aggregate quality metrics can fully mask a regression concentrated in one encoder.
• Model routing by query complexity (8B for simple lookups, 70B for complex reasoning) often saves more cost than any single infrastructure optimization.
• Encoders dominate correctness risk, decoders dominate cost - most of the engineering effort in a mature system shifts toward encoder quality and caching, even though most of the GPU spend stays on the decoder.

The counter-intuitive lesson is that the interesting distributed-systems problem in a multi-modal server is not the decoder - continuous batching and PagedAttention are solved problems borrowed wholesale from text-only serving. The real design work is upstream, in getting three different compute profiles to cooperate on one shared memory budget without any of them noticing the other two exist.

Frequently Asked Questions

Q: Why not run the vision and audio encoders on the same GPUs as the decoder to reduce node count? A: Encoders are compute-bound and benefit from batching many requests together; decode is latency-sensitive and needs predictable per-step timing. Colocating them means an image-heavy burst steals tensor core cycles from in-flight decodes, causing tail latency spikes exactly when load is highest.

Q: Why not just transcribe audio to text with ASR and skip audio tokens entirely? A: A transcript loses tone, emphasis, and non-speech audio cues the model could otherwise use, and running full ASR (encoder plus decoder) adds a second model’s latency in serial. Feeding raw encoder hidden states directly into the fused sequence keeps more signal and adds only encoder latency, not a full ASR pass.

Q: How much does the embedding cache actually save in practice? A: On production-like consumer traffic with a 30% repeat rate on media, it cuts blended cost per 1,000 requests by roughly 29% and cuts p99 TTFT by about 26%, since a hit skips both encoder GPU-seconds and the encoder’s micro-batching queue delay.

Q: How do you know the system is grounding correctly instead of just generating plausible-sounding text? A: A dedicated grounding guardrail checks whether objects and claims in the generated text are supported by the vision encoder’s detected content, an LLM-as-judge pass scores a golden eval set nightly, and both are sliced by modality mix so a regression in one encoder cannot hide behind aggregate metrics.

Q: Why use separate priority-based admission instead of strict first-in-first-out scheduling? A: FIFO admission lets one large multi-image request block a queue of cheap text-only requests behind it purely by arrival order. Priority-and-cost-aware admission fills any available KV cache headroom with cheaper requests first, keeping utilization high without starving interactive traffic.

Q: Why not cache full responses the way a CDN caches static assets? A: Multi-modal responses depend on the user’s specific question layered on top of the media, so two requests with the same image almost never produce the same answer. Caching the encoder’s output (the expensive, request-independent part) is where the real reuse lives, not the final generated text.

Interview Questions

Q: Design the admission control logic for a decoder that has to serve both text-only and multi-modal requests fairly under memory pressure. Expected depth: candidate should discuss KV cache block accounting per request, why token count alone underestimates multi-modal prefill cost, priority-and-cost-aware batch selection, and what happens when admission has to preempt an in-flight sequence.

Q: Walk through what happens, end to end, when a request contains two images and a text question referencing both. Expected depth: candidate should trace the request through modality routing with placeholder ordering, parallel vision encoder batching with cache lookups, token fusion preserving positional order, prefix cache matching, and streamed decode - and should flag where out-of-order encoder completion could misplace a media block if not handled carefully.

Q: How would you size the GPU footprint for a multi-modal server expecting 500 req/s with 50% of requests including one image? Expected depth: candidate should walk through KV cache-per-sequence memory math including visual token contribution, back into concurrent sequences per node, and separately size the vision encoder pool based on ViT batch throughput rather than assuming it scales with decoder node count.

Q: The grounding eval score looks healthy in aggregate but users are complaining about image misreads. How do you debug this? Expected depth: candidate should propose slicing the metric by modality mix and by specific image categories, checking for encoder version skew between the eval harness and production, and distinguishing an encoder regression from a fusion-ordering bug that misplaces visual tokens.

Q: Why might you choose to run two different decoder model sizes instead of one, and how would you route between them? Expected depth: candidate should discuss cost-latency tradeoffs of 8B versus 70B, routing signals (query length, explicit complexity hint, a lightweight classifier), and the risk of routing errors sending complex multi-modal reasoning to the smaller model and degrading quality silently.