A production RAG system doesn’t send a question to a model, it sends a question wrapped in everything the retriever thought might be relevant: eight, ten, sometimes fifteen retrieved chunks stitched into a single prompt that can easily run 8,000 tokens before the model has generated a single word of an answer. At 20,000 requests a minute, that is roughly 160 billion input tokens a day, and every one of them is billed whether or not the model actually needed it to answer the question. The problem is structurally similar to a trial lawyer handing a judge a 40-page brief when the judge has ten minutes and needs the same argument distilled to two pages without losing the one clause that wins the case. Cutting pages at random gets the length right and the argument wrong.

The naive fix is to just truncate: keep the first N tokens, or the last N, and drop the rest. Truncation is fast and free, but it treats every token as equally disposable, so it just as happily drops the one sentence containing the number the user actually asked about as it drops a paragraph of throat-clearing boilerplate. The next naive fix is to run the context through a smaller LLM and ask it to summarize, but that trades one large billed call for two: the summarization call itself burns tokens and adds a full round of decoding latency to a path that was supposed to get shorter, not slower. A third naive fix is a single fixed compression ratio applied to everything, but a customer support transcript compresses differently than a legal contract clause, and a ratio tuned for one silently mangles the other.

The forces in tension are cost, latency, and fidelity, and no two of them move together. Cost wants the context as small as possible. Latency wants the compression step itself to add almost nothing to a request that is already on a user-facing clock. Fidelity wants the compressed context to still contain whatever fact the answer actually depends on, and that fact is different on every request. Layered on top of all three is a fourth constraint that is easy to miss: if the serving layer downstream uses prefix or KV caching to avoid recomputing a repeated prompt prefix, a compression step that produces a slightly different token sequence on every request quietly destroys the cache hit rate it was never asked to protect.

We need to solve for three things simultaneously: how to decide which tokens matter without paying for a second full LLM call to make that decision, how to hit an exact token budget deterministically so identical input compresses to identical output and caching still works, and how to catch the rare case where compression silently removes the one detail an answer depends on before that mistake reaches a user.

Requirements and Constraints

Functional Requirements

• Given a raw context (retrieved chunks, conversation history, tool output) and a target token budget, produce a compressed version that fits the budget exactly, not approximately
• Score importance per token or per segment so the pruning decision correlates with what the downstream task actually needs, not just surface-level redundancy
• Support multiple compression policies keyed by content type (RAG context, multi-turn chat history, tool output) and select the right one per request
• Deduplicate near-identical retrieved chunks before scoring, since retrievers routinely surface the same fact from two or three overlapping source documents
• Never compress the stable prefix of a prompt, the system instructions and few-shot examples, that a downstream serving layer relies on for prefix or KV cache hits
• Run a fast faithfulness check on every compressed output and fall back to a less aggressive policy, or to the raw uncompressed context, when that check fails

Non-Functional Requirements

• Latency: compression adds no more than 150ms at p99 to a request’s critical path, since it sits between retrieval and generation on every request
• Throughput: sustain 20,000 requests/minute (about 333 req/s) of compression calls, absorbing bursts up to 4x baseline
• Quality: downstream task accuracy, measured via an eval harness, within 2 points of the uncompressed baseline at the chosen compression ratio
• Cost: net input token cost, including the scorer’s own compute cost, reduced by at least 50% relative to sending raw context
• Capacity: importance scorer footprint small enough to run on a shared pool of inference-class GPUs separate from the generation fleet, typically under 1GB of weights in FP16

Constraints

• Assume the downstream generation model is a hosted API billed per input token, at an assumed $2.50 per million input tokens for a mid-tier hosted model, and that this system controls only what gets sent to it, not how it is billed
• Assume retrieval has already happened; this system compresses the retrieved context and prior conversation turns, not the retrieval or ranking step itself
• Assume the importance scorer is a small encoder-only classifier, not the same large model doing generation, since running the generation model twice per request defeats the entire cost case
• Out of scope: fine-tuning the generation model to be compression-aware; the compressed prompt has to work with an unmodified downstream model
• Out of scope: compressing structured tool-call arguments or code, where every token is typically load-bearing; this design targets natural-language context

High-Level Architecture

The system has six major components. The Compression Policy Engine decides how aggressively to compress a given request based on content type and remaining cost budget. The Semantic Deduplicator collapses near-identical retrieved chunks before anything downstream has to consider them twice. The Token Importance Scorer assigns each token or span a score correlated with how much the eventual answer depends on it. The Coarse-to-Fine Pruning Engine uses those scores to remove content in two passes, first whole low-value chunks and then individual tokens, until the output hits an exact target budget. The Faithfulness Guardrail checks the compressed output against the original before it ships, and can veto the compression. The Cache-Aware Prompt Assembler stitches the compressed dynamic content behind an untouched, byte-identical prefix so a downstream serving layer’s prefix or KV cache still gets hit.

A request arrives with raw retrieved context and the current conversation turns. The Policy Engine looks up the content type and the remaining cost budget for this traffic segment and picks a target ratio and a latency ceiling. That policy, along with the raw context, flows into the compression pipeline proper: the Deduplicator collapses near-duplicate chunks, the Scorer assigns importance to what survives, and the Pruning Engine removes content in two passes until the token count matches the policy’s target exactly.

Before anything reaches the downstream model, the Faithfulness Guardrail runs a fast check against the original content. If it passes, the Cache-Aware Assembler stitches the compressed dynamic content behind the prompt’s stable prefix, which never enters the pipeline at all, and forwards the assembled prompt to the billed generation API. If the guardrail fails, the request loops back with a less aggressive policy, or with compression skipped entirely for that one request, since serving a correct answer at full cost beats serving a fast, cheap, wrong one.

The Compression Policy Engine

The policy engine’s job is to decide, before any scoring happens, how aggressively to compress a given request and how much latency that compression is allowed to spend.

The non-obvious part: a single global compression ratio is not a simplification, it is a bug. A support chat transcript is mostly filler and compresses well at 5x. A contract clause with three dates and two dollar amounts compresses badly at anything past 2x without losing a fact the answer depends on. A policy engine that ignores content type either wastes cost headroom on content that could compress harder, or destroys accuracy on content that can’t.

# Compression policy engine: selects target ratio and latency ceiling by content type
from dataclasses import dataclass
from enum import Enum

class ContentType(str, Enum):
    RAG_CONTEXT = "rag_context"
    CHAT_HISTORY = "chat_history"
    TOOL_OUTPUT = "tool_output"

@dataclass
class CompressionPolicy:
    policy_id: str
    target_ratio: float       # fraction of tokens to keep, e.g. 0.25 = compress to 25%
    max_latency_ms: int
    protect_entities: bool

DEFAULT_POLICIES: dict[ContentType, CompressionPolicy] = {
    ContentType.RAG_CONTEXT: CompressionPolicy("rag-default", target_ratio=0.25, max_latency_ms=150, protect_entities=True),
    ContentType.CHAT_HISTORY: CompressionPolicy("chat-default", target_ratio=0.20, max_latency_ms=100, protect_entities=False),
    ContentType.TOOL_OUTPUT: CompressionPolicy("tool-default", target_ratio=0.40, max_latency_ms=120, protect_entities=True),
}

def select_policy(content_type: ContentType, cost_budget_headroom: float) -> CompressionPolicy:
    base = DEFAULT_POLICIES[content_type]
    if cost_budget_headroom < 0.2:
        # under cost pressure, compress harder but never past a 15% floor
        tighter_ratio = max(0.15, base.target_ratio - 0.05)
        return CompressionPolicy(base.policy_id + "-tight", tighter_ratio, base.max_latency_ms, base.protect_entities)
    return base