A restaurant kitchen that only ever cooked one dish would be easy to run: one prep station, one burner schedule, one ticket rail. A kitchen that has to fire a seared steak, a delicate souffle, and a flash-boiled noodle dish off the same pass, on the same clock, with the same head chef, is a different problem entirely. Each dish has its own prep time, its own equipment, and its own tolerance for sitting under a heat lamp. That is the problem a multi-modal inference server has to solve: text prompts, images, and audio clips arrive on the same endpoint, need wildly different amounts of compute to turn into tokens, and all have to land in the same decoder so the model can reason across all three.

The scale numbers make the tension concrete. A production multi-modal endpoint serving a consumer assistant sees on the order of 250 requests per second blended across modalities. A single image, resized to 336x336 and patchified the way LLaVA and Qwen-VL do it, expands into 576 visual tokens before the decoder ever sees a word of the prompt. A 20-second audio clip, after a Whisper-style encoder compresses it, lands around 500 audio tokens. A plain text turn averages 300 tokens. Encoding an image on a vision tower takes 40-90ms of dense compute on a ViT; encoding audio takes longer because the encoder runs over the whole waveform sequentially. None of that overlaps for free with the autoregressive decode loop that has to hit a time-to-first-token (TTFT) budget under a second and keep generating at a steady time-per-output-token (TPOT).

The naive approach is to treat every request the same way: run the request through whatever encoder it needs, concatenate the output tokens with the text prompt, and hand the whole sequence to the decoder in the order it arrived. That falls apart in three places. First, image and audio encoding are throughput-bound batch operations that want to run many inputs at once on a GPU, while decode is latency-bound and wants small, fast steps; queuing them behind the same worker means a burst of images stalls every in-flight text-only decode. Second, the decoder’s KV cache has to hold 500-1000+ extra tokens per multi-modal request compared to a text-only one, so treating every request as equally expensive to admit blows the memory budget. Third, caching only helps if you cache the right thing at the right layer: caching a whole response is useless when the image changes every request, but caching the encoded representation of a repeated image (a product photo shown to 10,000 users) is enormously valuable.

We need to solve for heterogeneous compute scheduling across encoders and decoder, token-budget-aware admission control so multi-modal requests do not silently starve the KV cache, and modality-specific caching so repeated media never gets re-encoded. Get those three right and the rest of the system - routing, batching, streaming - falls into place around them.

Requirements and Constraints

Functional Requirements

• Accept requests containing any combination of a text prompt, one or more images, and one audio clip, and produce a single coherent streamed text response.
• Detect the modality mix per request and route each non-text input to the correct encoder (vision tower for images, audio encoder for speech/sound) before any decoding starts.
• Fuse encoder outputs and text tokens into a single ordered sequence the decoder can attend over, preserving positional relationships between the media and the surrounding text.
• Stream tokens back to the client over SSE as soon as the first token is available, regardless of which modalities were present.
• Cache encoder outputs for repeated media (same image bytes, same audio hash) so identical inputs never pay encoding cost twice.
• Support at least two model sizes (a fast 8B-class decoder and a stronger 70B-class decoder) with routing based on request complexity or explicit client hint.

Non-Functional Requirements

• Latency: p50 TTFT under 400ms for text-only requests, p99 TTFT under 900ms when an image or audio encode is on the critical path; TPOT under 35ms per token at p50.
• Throughput: sustain 250 requests/second blended traffic, with peak bursts to 600 requests/second for 2-3 minutes during traffic spikes.
• Quality: cross-modal grounding accuracy (does the generated text correctly reference the image/audio content) above 92% on a held-out eval set; hallucinated object references under 3%.
• Cost: median cost per 1,000 requests under $4.50 blended across modalities; image and audio encoding must not add more than 25% to the per-request GPU-second cost of an equivalent text-only request.
• Capacity: KV cache budget per node must reserve headroom for the 99th-percentile multi-modal token count (roughly 1,100 fused tokens of context before generation starts), not just the text-only average.

Constraints

• Decoder: a single family of instruction-tuned multi-modal-capable LLMs (8B and 70B variants) served with vLLM-style continuous batching and PagedAttention.
• Vision tower: a frozen ViT-L/14-class encoder producing 576 tokens per 336x336 image tile, projected into the decoder’s embedding space via a learned adapter.
• Audio encoder: a Whisper-large-class encoder producing a compressed token stream at roughly 25 tokens per second of audio.
• Hardware: 8x A100 80GB GPU nodes, with dedicated node pools for encoders versus decoder rather than colocating everything on one GPU.
• Out of scope: image or audio generation (this server only consumes media, it does not produce it), video input, and real-time bidirectional voice (this is turn-based, not a live voice call).

High-Level Architecture

The system has six major components: an ingress gateway that accepts and validates multi-modal requests, a modality router that inspects each request and dispatches non-text payloads to the right encoder pool, a vision encoder pool and an audio encoder pool that each turn raw media into token sequences on dedicated GPUs, a token fuser that merges encoder outputs with text tokens and manages a prefix cache for repeated media, and a decoder serving pool that runs continuous batching and PagedAttention over the fused sequences and streams tokens back to the client.

A request lands at the ingress gateway carrying a text prompt and, optionally, one or more image blobs and an audio blob. The gateway validates payload sizes, strips EXIF and other metadata from images, and forwards the request to the modality router with a manifest describing what’s present. The router does not do any encoding itself - its only job is to fan out the image and audio payloads to their respective encoder pools while the text prompt waits in the token fuser. Each encoder pool batches inputs across requests (many images from different users can share one forward pass through the ViT), checks the embedding cache first, and returns a token sequence plus positional metadata back to the token fuser.

The token fuser is the seam between “encoding” and “decoding.” It assembles the final input sequence - text tokens, visual tokens, audio tokens, in the order the client specified - and hands the completed sequence to the decoder pool for prefill and generation. The decoder pool treats a fused multi-modal sequence exactly like a long text sequence for scheduling purposes: it goes through the same continuous batching admission logic, occupies PagedAttention blocks in the KV cache, and streams tokens back through the gateway over SSE.

The Modality Router

The modality router’s job is to look at a request’s manifest and decide, in under a millisecond, which encoder pools need to see this request and in what order their outputs must be stitched back together.