A production RAG system doesn’t send a question to a model, it sends a question wrapped in everything the retriever thought might be relevant: eight, ten, sometimes fifteen retrieved chunks stitched into a single prompt that can easily run 8,000 tokens before the model has generated a single word of an answer. At 20,000 requests a minute, that is roughly 160 billion input tokens a day, and every one of them is billed whether or not the model actually needed it to answer the question. The problem is structurally similar to a trial lawyer handing a judge a 40-page brief when the judge has ten minutes and needs the same argument distilled to two pages without losing the one clause that wins the case. Cutting pages at random gets the length right and the argument wrong.

The naive fix is to just truncate: keep the first N tokens, or the last N, and drop the rest. Truncation is fast and free, but it treats every token as equally disposable, so it just as happily drops the one sentence containing the number the user actually asked about as it drops a paragraph of throat-clearing boilerplate. The next naive fix is to run the context through a smaller LLM and ask it to summarize, but that trades one large billed call for two: the summarization call itself burns tokens and adds a full round of decoding latency to a path that was supposed to get shorter, not slower. A third naive fix is a single fixed compression ratio applied to everything, but a customer support transcript compresses differently than a legal contract clause, and a ratio tuned for one silently mangles the other.

The forces in tension are cost, latency, and fidelity, and no two of them move together. Cost wants the context as small as possible. Latency wants the compression step itself to add almost nothing to a request that is already on a user-facing clock. Fidelity wants the compressed context to still contain whatever fact the answer actually depends on, and that fact is different on every request. Layered on top of all three is a fourth constraint that is easy to miss: if the serving layer downstream uses prefix or KV caching to avoid recomputing a repeated prompt prefix, a compression step that produces a slightly different token sequence on every request quietly destroys the cache hit rate it was never asked to protect.

We need to solve for three things simultaneously: how to decide which tokens matter without paying for a second full LLM call to make that decision, how to hit an exact token budget deterministically so identical input compresses to identical output and caching still works, and how to catch the rare case where compression silently removes the one detail an answer depends on before that mistake reaches a user.

Requirements and Constraints

Functional Requirements

• Given a raw context (retrieved chunks, conversation history, tool output) and a target token budget, produce a compressed version that fits the budget exactly, not approximately
• Score importance per token or per segment so the pruning decision correlates with what the downstream task actually needs, not just surface-level redundancy
• Support multiple compression policies keyed by content type (RAG context, multi-turn chat history, tool output) and select the right one per request
• Deduplicate near-identical retrieved chunks before scoring, since retrievers routinely surface the same fact from two or three overlapping source documents
• Never compress the stable prefix of a prompt, the system instructions and few-shot examples, that a downstream serving layer relies on for prefix or KV cache hits
• Run a fast faithfulness check on every compressed output and fall back to a less aggressive policy, or to the raw uncompressed context, when that check fails

Non-Functional Requirements

• Latency: compression adds no more than 150ms at p99 to a request’s critical path, since it sits between retrieval and generation on every request
• Throughput: sustain 20,000 requests/minute (about 333 req/s) of compression calls, absorbing bursts up to 4x baseline
• Quality: downstream task accuracy, measured via an eval harness, within 2 points of the uncompressed baseline at the chosen compression ratio
• Cost: net input token cost, including the scorer’s own compute cost, reduced by at least 50% relative to sending raw context
• Capacity: importance scorer footprint small enough to run on a shared pool of inference-class GPUs separate from the generation fleet, typically under 1GB of weights in FP16

Constraints

• Assume the downstream generation model is a hosted API billed per input token, at an assumed $2.50 per million input tokens for a mid-tier hosted model, and that this system controls only what gets sent to it, not how it is billed
• Assume retrieval has already happened; this system compresses the retrieved context and prior conversation turns, not the retrieval or ranking step itself
• Assume the importance scorer is a small encoder-only classifier, not the same large model doing generation, since running the generation model twice per request defeats the entire cost case
• Out of scope: fine-tuning the generation model to be compression-aware; the compressed prompt has to work with an unmodified downstream model
• Out of scope: compressing structured tool-call arguments or code, where every token is typically load-bearing; this design targets natural-language context

High-Level Architecture

The system has six major components. The Compression Policy Engine decides how aggressively to compress a given request based on content type and remaining cost budget. The Semantic Deduplicator collapses near-identical retrieved chunks before anything downstream has to consider them twice. The Token Importance Scorer assigns each token or span a score correlated with how much the eventual answer depends on it. The Coarse-to-Fine Pruning Engine uses those scores to remove content in two passes, first whole low-value chunks and then individual tokens, until the output hits an exact target budget. The Faithfulness Guardrail checks the compressed output against the original before it ships, and can veto the compression. The Cache-Aware Prompt Assembler stitches the compressed dynamic content behind an untouched, byte-identical prefix so a downstream serving layer’s prefix or KV cache still gets hit.

A request arrives with raw retrieved context and the current conversation turns. The Policy Engine looks up the content type and the remaining cost budget for this traffic segment and picks a target ratio and a latency ceiling. That policy, along with the raw context, flows into the compression pipeline proper: the Deduplicator collapses near-duplicate chunks, the Scorer assigns importance to what survives, and the Pruning Engine removes content in two passes until the token count matches the policy’s target exactly.

Before anything reaches the downstream model, the Faithfulness Guardrail runs a fast check against the original content. If it passes, the Cache-Aware Assembler stitches the compressed dynamic content behind the prompt’s stable prefix, which never enters the pipeline at all, and forwards the assembled prompt to the billed generation API. If the guardrail fails, the request loops back with a less aggressive policy, or with compression skipped entirely for that one request, since serving a correct answer at full cost beats serving a fast, cheap, wrong one.

The Compression Policy Engine

The policy engine’s job is to decide, before any scoring happens, how aggressively to compress a given request and how much latency that compression is allowed to spend.

The non-obvious part: a single global compression ratio is not a simplification, it is a bug. A support chat transcript is mostly filler and compresses well at 5x. A contract clause with three dates and two dollar amounts compresses badly at anything past 2x without losing a fact the answer depends on. A policy engine that ignores content type either wastes cost headroom on content that could compress harder, or destroys accuracy on content that can’t.

# Compression policy engine: selects target ratio and latency ceiling by content type
from dataclasses import dataclass
from enum import Enum

class ContentType(str, Enum):
    RAG_CONTEXT = "rag_context"
    CHAT_HISTORY = "chat_history"
    TOOL_OUTPUT = "tool_output"

@dataclass
class CompressionPolicy:
    policy_id: str
    target_ratio: float       # fraction of tokens to keep, e.g. 0.25 = compress to 25%
    max_latency_ms: int
    protect_entities: bool

DEFAULT_POLICIES: dict[ContentType, CompressionPolicy] = {
    ContentType.RAG_CONTEXT: CompressionPolicy("rag-default", target_ratio=0.25, max_latency_ms=150, protect_entities=True),
    ContentType.CHAT_HISTORY: CompressionPolicy("chat-default", target_ratio=0.20, max_latency_ms=100, protect_entities=False),
    ContentType.TOOL_OUTPUT: CompressionPolicy("tool-default", target_ratio=0.40, max_latency_ms=120, protect_entities=True),
}

def select_policy(content_type: ContentType, cost_budget_headroom: float) -> CompressionPolicy:
    base = DEFAULT_POLICIES[content_type]
    if cost_budget_headroom < 0.2:
        # under cost pressure, compress harder but never past a 15% floor
        tighter_ratio = max(0.15, base.target_ratio - 0.05)
        return CompressionPolicy(base.policy_id + "-tight", tighter_ratio, base.max_latency_ms, base.protect_entities)
    return base

The analogy is a newspaper’s style guide: the front-page lead gets 1,200 words, a photo caption gets 20, and nobody uses the caption’s length limit for the lead. What breaks if this is simplified to one ratio for everything: the content type that most needs its detail preserved, usually the one with numbers and named entities, gets squeezed exactly as hard as the content type that is mostly padding.

The Semantic Deduplicator

The deduplicator’s job is to collapse near-identical retrieved chunks into one representative chunk before the scorer ever has to consider them.

The non-obvious part: exact-text deduplication catches almost nothing in practice. Two chunks pulled from different source documents rarely share the same sentence verbatim, but they routinely restate the same fact in different words, and a scorer that treats both as independently important wastes budget the pruning engine can’t get back later.

# Semantic deduplicator: greedily clusters near-duplicate chunks by embedding cosine similarity
import numpy as np

def deduplicate_chunks(chunks: list[str], embeddings: np.ndarray, similarity_threshold: float = 0.92) -> list[int]:
    # embeddings[i] corresponds to chunks[i], already L2-normalized
    n = len(chunks)
    kept: list[int] = []
    suppressed = np.zeros(n, dtype=bool)

    # process in retrieval-rank order so the highest-ranked chunk in a cluster survives
    for i in range(n):
        if suppressed[i]:
            continue
        kept.append(i)
        sims = embeddings[i] @ embeddings[i + 1:].T
        for offset, sim in enumerate(sims):
            j = i + 1 + offset
            if not suppressed[j] and sim >= similarity_threshold:
                suppressed[j] = True
    return kept

The analogy is a research assistant noticing that three sources cite the same statistic and keeping just one, with a note that it was corroborated. What breaks without this step: the scorer and pruning engine spend part of the 2,000-token budget representing the same fact three times, which means unique facts elsewhere in the context lose out on budget they should have gotten instead.

The Token Importance Scorer

The scorer’s job is to assign every surviving token, or a small span of tokens, a score that reflects how much the eventual answer depends on it.

The non-obvious part: this does not require a full LLM call. A small encoder-only classifier, trained via distillation against a larger model’s judgments of which tokens are safe to drop, gets most of the accuracy of an LLM-based judgment at a small fraction of the compute, because the task is binary classification, not generation. This is the approach behind Microsoft’s LLMLingua-2, which trains an XLM-RoBERTa-large-class token classifier on data labeled by asking a large LLM which tokens it could remove from a passage without changing the answer to a downstream question.

# Token importance scorer: batched binary keep/drop classification over 512-token windows
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

class ImportanceScorer:
    def __init__(self, model_name: str = "microsoft/llmlingua-2-xlm-roberta-large", device: str = "cuda:0"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForTokenClassification.from_pretrained(model_name).to(device).eval()
        self.device = device
        self.window = 512

    @torch.inference_mode()
    def score(self, text: str) -> list[tuple[str, float]]:
        encoded = self.tokenizer(text, return_offsets_mapping=True, truncation=False)
        input_ids = encoded["input_ids"]
        tokens = self.tokenizer.convert_ids_to_tokens(input_ids)

        scores: list[float] = []
        for start in range(0, len(input_ids), self.window):
            window_ids = input_ids[start:start + self.window]
            batch = torch.tensor([window_ids], device=self.device)
            logits = self.model(batch).logits  # [1, seq_len, 2] -> keep vs drop
            probs = torch.softmax(logits, dim=-1)[0, :, 1]  # P(keep)
            scores.extend(probs.tolist())
        return list(zip(tokens, scores))

The analogy is a paralegal who has learned, from watching a partner mark up a hundred briefs in red ink, which sentences a partner keeps and which ones get crossed out, without needing the partner to review every single page personally. What breaks if this is simplified to a rule-based heuristic, like dropping stopwords or short sentences: it correctly removes obvious filler but has no signal about which content-bearing sentences are actually load-bearing for a specific question, which is exactly the judgment that matters.

The Coarse-to-Fine Pruning Engine

The pruning engine’s job is to take the scorer’s importance values and an exact target token count, and produce output that hits that count precisely, not approximately.

The non-obvious part: pruning at the token level in a single pass, sorted purely by score, produces incoherent fragments, because it can keep token 4 and token 400 of the same sentence while dropping everything in between. A two-stage pass, coarse at the chunk level and fine at the token level within the chunks that survive, preserves local coherence while still landing on an exact number.

# Coarse-to-fine pruning: drop whole low-value chunks first, then trim tokens within survivors
from dataclasses import dataclass

@dataclass
class Chunk:
    chunk_id: int
    tokens: list[str]
    token_scores: list[float]

    @property
    def avg_score(self) -> float:
        return sum(self.token_scores) / max(1, len(self.token_scores))

def prune(chunks: list[Chunk], target_tokens: int) -> list[str]:
    # coarse stage: rank whole chunks by average importance, keep chunks until
    # the running total would exceed target, so the fine stage has less to trim
    ranked = sorted(chunks, key=lambda c: (-c.avg_score, c.chunk_id))  # deterministic tie-break
    survivors: list[Chunk] = []
    running_total = 0
    for chunk in ranked:
        if running_total + len(chunk.tokens) <= target_tokens * 1.3:
            survivors.append(chunk)
            running_total += len(chunk.tokens)

    survivors.sort(key=lambda c: c.chunk_id)  # restore original document order

    # fine stage: trim individual tokens within surviving chunks until the exact budget is hit
    scored_tokens = [
        (chunk.chunk_id, position, token, score)
        for chunk in survivors
        for position, (token, score) in enumerate(zip(chunk.tokens, chunk.token_scores))
    ]
    keep_count = min(target_tokens, len(scored_tokens))
    # deterministic tie-break on (chunk_id, position) so identical input always
    # produces an identical output, which matters for downstream prefix caching
    ranked_tokens = sorted(scored_tokens, key=lambda t: (-t[3], t[0], t[1]))[:keep_count]
    keep_set = {(c, p) for c, p, _, _ in ranked_tokens}

    result: list[str] = []
    for chunk_id, position, token, _ in sorted(scored_tokens, key=lambda t: (t[0], t[1])):
        if (chunk_id, position) in keep_set:
            result.append(token)
    return result

The analogy is packing a suitcase: decide which whole bags stay home first, then trim within the bags you’re actually bringing, rather than pulling individual socks out of every bag at random until the scale reads the right weight. What breaks if this is simplified to a single token-level sort across the entire context: it destroys sentence structure unpredictably, and a downstream model that receives grammatically broken input performs worse than the token count alone would predict.

The Faithfulness Guardrail

The guardrail’s job is to catch the rare case where compression likely removed something the answer depends on, before the compressed prompt ever reaches the billed generation call.

# Faithfulness guardrail: fast checks that must pass before a compressed prompt ships
import re

PROTECTED_PATTERNS = [
    re.compile(r"\b\d[\d,]*\.?\d*\b"),          # numbers, dollar amounts, dates
    re.compile(r"\b(not|no|never|without)\b", re.IGNORECASE),  # negations
    re.compile(r'"[^"]{3,}"'),                   # quoted spans
]

def faithfulness_check(raw_text: str, compressed_tokens: list[str], embed_fn) -> dict:
    compressed_text = " ".join(compressed_tokens)

    dropped_protected = 0
    for pattern in PROTECTED_PATTERNS:
        raw_matches = set(pattern.findall(raw_text))
        compressed_matches = set(pattern.findall(compressed_text))
        dropped_protected += len(raw_matches - compressed_matches)

    raw_vec, compressed_vec = embed_fn(raw_text), embed_fn(compressed_text)
    semantic_recall = float(raw_vec @ compressed_vec)  # cosine sim, both pre-normalized

    passed = dropped_protected == 0 and semantic_recall >= 0.90
    return {
        "dropped_protected_spans": dropped_protected,
        "semantic_recall": semantic_recall,
        "pass": passed,
    }

The analogy is a proofreader running a final checklist against the original draft, not trusting that the editor’s cuts were safe just because the page count came out right. What breaks without this step: compression that is correct on average still fails on individual requests, and nothing downstream would ever notice until a user reports a wrong answer that traces back to a dropped clause.

The Cache-Aware Prompt Assembler

The assembler’s job is to combine the compressed dynamic context with the prompt’s untouched stable prefix, in a way that a downstream serving layer’s prefix or KV cache still recognizes as a repeated prefix.

// Cache-aware assembler: stitches compressed context behind an untouched, hashed prefix
package assembler

import (
	"crypto/sha256"
	"encoding/hex"
)

type AssembledPrompt struct {
	PrefixHash    string
	FullPrompt    string
	PrefixTokens  int
	DynamicTokens int
}

// stablePrefix (system instructions, few-shot examples) must never be touched by
// compression. It is hashed so the serving layer can key its prefix cache on it.
func Assemble(stablePrefix string, compressedContext string, prefixTokenCount, dynamicTokenCount int) AssembledPrompt {
	hash := sha256.Sum256([]byte(stablePrefix))
	return AssembledPrompt{
		PrefixHash:    hex.EncodeToString(hash[:]),
		FullPrompt:    stablePrefix + "\n" + compressedContext,
		PrefixTokens:  prefixTokenCount,
		DynamicTokens: dynamicTokenCount,
	}
}

Put together, the pipeline’s job is narrower than it sounds: it never touches the part of the prompt that caching depends on, and it makes the part it does touch behave deterministically enough that caching keeps working on the parts around it.

Data Model

The system tracks three kinds of state: relational policy and run metadata, a content-hash memoization cache for repeated chunks, and high-volume compression telemetry events.

-- Compression policy: target ratio and latency ceiling per content type
CREATE TABLE compression_policies (
    policy_id         TEXT PRIMARY KEY,
    content_type      TEXT NOT NULL CHECK (content_type IN ('rag_context', 'chat_history', 'tool_output')),
    target_ratio      NUMERIC(3,2) NOT NULL CHECK (target_ratio > 0 AND target_ratio <= 1),
    max_latency_ms    INT NOT NULL,
    protect_entities  BOOLEAN NOT NULL DEFAULT true,
    active            BOOLEAN NOT NULL DEFAULT true,
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Compression run: one row per compressed segment, keyed for memoization on content hash
CREATE TABLE compression_runs (
    run_id              UUID PRIMARY KEY,
    content_hash        TEXT NOT NULL,
    policy_id           TEXT NOT NULL REFERENCES compression_policies(policy_id),
    raw_tokens           INT NOT NULL,
    compressed_tokens    INT NOT NULL,
    faithfulness_score  NUMERIC(4,3),
    status              TEXT NOT NULL CHECK (status IN ('ok', 'fallback_raw', 'fallback_less_aggressive')),
    created_at          TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX idx_compression_runs_hash ON compression_runs (content_hash);
CREATE INDEX idx_compression_runs_policy ON compression_runs (policy_id, created_at DESC);

Compression telemetry is high-volume and emitted on every request, so it is modeled as an event rather than a row that gets updated:

// Compression event: emitted by the pipeline on every request's compression decision
syntax = "proto3";

package aisd.promptcompression;

message CompressionEvent {
  string request_id = 1;
  string policy_id = 2;
  int32 raw_tokens = 3;
  int32 compressed_tokens = 4;
  double compression_latency_ms = 5;
  double faithfulness_score = 6;
  bool cache_hit = 7;
  bool guardrail_fallback = 8;
  int64 timestamp_ms = 9;
}

The memoization cache and the prefix boundary registry both need sub-millisecond lookups on the request path, so both live in Redis rather than the relational store:

HSET compressed_chunk:9f2a1c tokens 480 ratio 0.25 policy rag-default
EXPIRE compressed_chunk:9f2a1c 86400
SETEX prefix_boundary:session-4471 3600 "sys_prompt_v3::few_shot_v2"

The partitioning key is content_hash for compression_runs, since the whole point of that table is memoizing identical chunks across requests, and policy_id for compression_policies, since policy lookups happen once per request and should never require a table scan.

Key Algorithms and Protocols

Distilled Token Classification for Importance Scoring

Already covered in the Token Importance Scorer section above. Time complexity is linear in token count per window, O(w) per 512-token window, and the edge case worth naming is a context shorter than one window: the scorer should still run rather than being skipped, since even short contexts routinely contain droppable filler.

Coarse-to-Fine Budget Allocation

Already covered in the Coarse-to-Fine Pruning Engine section above. The coarse stage is O(n log n) to sort chunks by average score, and the fine stage is O(m log m) to sort the surviving tokens, where m is much smaller than the original token count after the coarse pass has already dropped whole low-value chunks. The edge case: if the target budget exceeds the raw token count, pruning is a no-op and the pipeline should short-circuit rather than run the scorer for nothing.

Embedding-Based Near-Duplicate Clustering

Already covered in the Semantic Deduplicator section above. The naive pairwise comparison is O(n^2) in the number of chunks, which is fine at the scale of a handful of retrieved chunks per request but should move to an approximate nearest-neighbor index if a future version needs to deduplicate across hundreds of chunks at once. The edge case is threshold sensitivity: a similarity threshold set too low merges genuinely distinct facts that happen to share vocabulary, and too high leaves near-duplicates unmerged, wasting budget on redundant content.

Deterministic Prefix Boundary Detection

The cache-aware assembler needs to know, for every request, exactly where the stable prefix ends and the dynamic context begins. This is a fixed offset for a given deployment, not something computed per request, because the whole point is that the prefix never changes shape while a given system prompt and few-shot set are live. The edge case is a mid-rollout deploy that changes the system prompt: every in-flight request must resolve to one prefix version or the other, never a mix, or the prefix hash a serving layer’s cache expects will not match either version.

Scaling and Performance

A single importance scorer replica’s throughput does not scale past its GPU’s forward-pass capacity, so what scales is the number of scorer replicas behind a load-balanced pool, sharded by request so no single GPU becomes a queueing bottleneck during a burst.

Given:
  - 333 req/s (20,000 req/min), avg 8,000 raw context tokens per request
  - Importance scorer: XLM-RoBERTa-large-class token classifier, 512-token max window
  - Windows per request: ceil(8,000 / 512) = 16
  - Scorer throughput per A10 GPU (batch 32, fp16): ~1,200 windows/sec

Windows/sec needed: 333 * 16 = 5,328
GPUs needed: ceil(5,328 / 1,200) = 5, plus headroom for a 4x burst -> 20 GPUs at peak

Compressed output: 2,000 tokens/request -> compression ratio 4x, 75% token reduction
Token throughput removed before billing: 333 req/s * (8,000 - 2,000) tokens
  = 1,998,000 tokens/s that never reach the billed LLM API

The scorer’s memory footprint is small enough, under 1.2GB in FP16, that a single A10 comfortably runs multiple replicas, so the binding constraint is throughput, not capacity, unlike a generation model where GPU memory for weights and KV cache is usually the ceiling. The content-hash memoization cache absorbs a meaningful share of this load in practice: with roughly 40% of retrieved chunks repeating across queries that hit the same popular source documents, close to 40% of what would otherwise be scorer traffic resolves from a cache hit instead.

Cost and Token Economics

The dollar case rests almost entirely on how much of the 8,000-token baseline actually needs to reach the billed generation API, not on any particular cleverness in how the scorer works.

Configuration	Input tokens billed/req	Cost/1,000 req at $2.50/M tokens	Scorer compute/1,000 req	Downstream accuracy
Raw pass-through, no compression	8,000	$20.00	$0.00	Baseline (100%)
Naive fixed truncation to 2,000 tokens	2,000	$5.00	$0.00	Baseline minus 8-12 points
Classifier compression to 2,000 tokens (chosen)	2,000	$5.00	$0.35	Baseline minus 1-2 points
Classifier compression + cache-aware prefix reuse	2,300 (300 cached + 2,000 dynamic)	$3.20 effective, cached prefix billed at a discounted cache-read rate	$0.35	Baseline minus 1-2 points

The measured optimization is the combination row: moving from raw pass-through to classifier compression with a cache-aware prefix drops effective cost per 1,000 requests from $20.00 to roughly $3.20, an 84% reduction, while naive truncation gets a similar token count at a similar sticker price but gives up 8 to 12 points of downstream accuracy to get there. The gap between naive truncation and classifier compression is the entire argument for building the scorer at all; both hit the same token budget, only one of them hits it without breaking the answer.

Quality, Evaluation, and Guardrails

The quality question is not whether the generation model’s outputs are good in the abstract, it is whether a compressed prompt leads the model to the same answer a raw prompt would have, since compression that silently changes the answer fails softly and looks identical to a correct response until someone checks.

# Offline eval: compares answers on compressed vs raw context across a golden Q&A set
def evaluate_compression_policy(golden_set: list[dict], answer_fn, judge_fn) -> dict:
    # golden_set entries: {"question": str, "raw_context": str, "compressed_context": str, "reference_answer": str}
    matches = 0
    for item in golden_set:
        raw_answer = answer_fn(item["question"], item["raw_context"])
        compressed_answer = answer_fn(item["question"], item["compressed_context"])
        # judge_fn returns True if compressed_answer is materially equivalent to raw_answer
        # against the same reference, using an LLM-as-judge prompt with the reference answer
        if judge_fn(item["reference_answer"], raw_answer, compressed_answer):
            matches += 1
    accuracy_delta = (len(golden_set) - matches) / len(golden_set)
    return {"total": len(golden_set), "matched": matches, "accuracy_delta_pct": accuracy_delta * 100}

Offline, every candidate policy runs against a fixed golden Q&A set before it is promoted, comparing answers generated from compressed context against answers from raw context, both judged against the same reference answer by an LLM-as-judge prompt. Online, the signal is the guardrail’s fallback rate and the regeneration rate on responses served from compressed prompts. The metric that gates a policy’s rollout is accuracy delta versus the raw-context baseline, and the threshold is 2 percentage points; a policy that regresses further than that on the golden set does not ship, regardless of how much it saves in tokens.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Scorer misclassifies a load-bearing token as droppable	faithfulness guardrail semantic recall or protected-pattern check fails	compressed prompt would produce a wrong answer	guardrail blocks the compressed version, falls back to raw context or a less aggressive policy for that request
Compression latency spike from GPU contention on the scorer pool	p99 compression latency alert crosses the 150ms budget	request risks breaching its end-to-end SLA	bypass compression for that request, send raw context, eat the token cost rather than block
Semantic deduplicator over-merges distinct facts sharing vocabulary	dedup ratio anomaly monitor flags an unusual spike in merged chunks	unique facts get treated as redundant and dropped from consideration entirely	raise the similarity threshold, replay affected traffic against the eval harness before re-lowering it
Prefix cache invalidation from non-deterministic pruning output	downstream serving layer’s prefix cache hit rate drops unexpectedly	every request recomputes the prefix, losing the caching benefit entirely	pin deterministic tie-breaking in the pruning engine, audit for any remaining source of nondeterminism
Faithfulness guardrail false negative on a dropped negation	near-miss surfaces in LLM-as-judge grading during a routine eval run, not before	a compressed prompt inverts a sentence’s meaning without tripping a protected-pattern match	add negation-specific protected patterns, treat any negation token as always-protected regardless of score
Scorer version skew mid-rollout across concurrent requests	request-level scorer version tag mismatch against the expected cohort version	inconsistent compression behavior across requests issued moments apart	pin scorer version per request cohort at policy-selection time, never mix versions within one in-flight request

Comparison of Approaches

Approach	Latency overhead	Cost/1,000 req	Complexity	Failure mode	Best fit
No compression, raw pass-through	None	$20.00	Low	None from compression; pure cost and context-window pressure at scale	Low-volume traffic or contexts already well under the token budget
Fixed truncation (keep first or last N tokens)	Negligible	$5.00	Low	Drops content based on position, indifferent to what the answer needs	Contexts with a known, reliable “most important part goes first/last” structure
Abstractive LLM summarization	High, a full second LLM call and decode pass	$6.50-8.00 (summarization call itself burns tokens)	Medium	Summarization can hallucinate or omit specifics the original preserved	Long-form content review where latency is not on a user-facing critical path
Extractive embedding-based sentence selection (RECOMP-style)	Low-moderate	$5.50	Medium	Selects whole sentences, coarser control than token-level budget matching	Contexts where sentence-level granularity is acceptable and exact budgets matter less
Classifier-based token compression (LLMLingua-2-style, chosen)	Low, under 150ms p99	$5.00-5.35 (incl. scorer compute)	High	A misclassified token can silently drop task-relevant content without a guardrail	Latency-sensitive, high-volume production RAG and chat traffic with a hard cost target

For production RAG traffic with a real latency SLA and a real token budget, classifier-based compression paired with a faithfulness guardrail is the right default: it gets within 150ms of raw pass-through while cutting cost by roughly 75%, and it catches the failure mode that makes naive truncation risky. Teams with low volume or contexts that are already comfortably under budget can reasonably skip compression entirely, since the added complexity buys the least when there is little cost pressure to relieve.

Key Takeaways

• Classifier-based token scoring gets most of the accuracy of an LLM judgment at a fraction of the compute, because deciding what to drop is a classification problem, not a generation problem.
• Coarse-to-fine pruning preserves local coherence by dropping whole low-value chunks before trimming individual tokens, instead of sorting the entire context by score in one flat pass.
• Deterministic tie-breaking in the pruning engine is not a nice-to-have, it is what keeps identical inputs producing identical outputs, which downstream prefix caching depends on completely.
• The stable prompt prefix must never enter the compression pipeline at all, or every request pays the cost of recomputing a prefix a serving layer’s cache was supposed to make free.
• A faithfulness guardrail is the difference between compression that is correct on average and compression that is safe on every request, and those are not the same property.
• Semantic deduplication before scoring prevents the same fact, restated across multiple retrieved chunks, from competing for compression budget against itself.
• Content-hash memoization turns repeated retrieval hits into free compression, since a chunk scored and pruned once for a given policy never needs the scorer’s compute again.
• Naive truncation and classifier compression can hit the identical token budget and cost the identical amount, and the entire case for building a scorer is the accuracy gap between them.

The counter-intuitive lesson is that the expensive-looking part of this system, running a model to decide what to cut, is cheaper than the cheap-looking part, a fixed truncation rule, once accuracy loss is priced in. A truncation rule that quietly costs 10 points of task accuracy is not actually the cheap option; it just moved the cost somewhere the token bill doesn’t show it.

Frequently Asked Questions

Q: Why not just truncate to the last N tokens instead of building a scoring pipeline? A: Truncation is free and instant, but it drops content based on position, not relevance, so it removes the fact an answer depends on exactly as readily as it removes filler. In practice this costs 8 to 12 points of downstream accuracy at the same token budget a scored approach achieves for about $0.35 per 1,000 requests in added compute.

Q: Why not run a smaller LLM to summarize the context instead of a lightweight classifier? A: Summarization requires a full second generation call, which burns its own input and output tokens and adds a complete decode pass to the latency budget, often pushing total added latency well past the 150ms ceiling this design targets. A classifier only needs a forward pass, no decoding, which is why it stays fast enough to sit on the critical path of every request.

Q: What is the actual dollar impact of this pipeline at scale? A: At $2.50 per million input tokens and 20,000 requests per minute, moving from an 8,000-token raw context to a 2,300-token compressed-plus-cached-prefix prompt cuts cost per 1,000 requests from $20.00 to roughly $3.20, an 84% reduction, after accounting for the scorer’s own compute cost.

Q: How do you know compression hasn’t silently dropped something that changes the answer? A: Every compressed output passes through a faithfulness guardrail that checks protected token patterns, numbers, negations, quoted spans, and a semantic recall score against the original before it ships. Anything that fails falls back to a less aggressive policy or to the raw context rather than reaching the generation call unchecked.

Q: Does compressing the context defeat prefix or KV caching on the serving layer? A: Only if compression touches the stable prefix, which this design never does. The compression pipeline operates exclusively on the dynamic context appended after the prefix, and deterministic pruning ensures identical dynamic content always compresses to the identical token sequence, so the prefix cache keeps hitting normally.

Q: Is it safe to hot-swap the importance scorer model mid-traffic? A: Not without pinning a version per request cohort. If concurrent in-flight requests resolve against different scorer versions, compression behavior becomes inconsistent in ways that are hard to attribute to any single cause. A rolling deploy should route each new request to one scorer version deterministically rather than letting version selection race with traffic.

Interview Questions

Q: Walk through what happens to a 8,000-token RAG context from the moment it’s retrieved to the moment it reaches the generation API. Expected depth: candidate should trace deduplication removing near-identical chunks, the scorer assigning per-token importance, the pruning engine’s coarse chunk-level pass followed by a fine token-level pass to hit an exact budget, the faithfulness guardrail’s check, and the assembler stitching the result behind an untouched cached prefix. Should be able to name where each stage’s latency and cost come from.

Q: Why does a two-stage coarse-to-fine pruning approach beat a single-pass token-level sort by importance score? Expected depth: should explain that a flat token-level sort ignores document structure and can keep isolated tokens from the middle of a sentence while dropping tokens around them, producing incoherent input. A coarse chunk-level pass preserves local coherence in whatever chunks survive, and the fine pass only needs to trim within already-coherent units.

Q: How would you detect that a compression policy has quietly regressed in production, and what would you do about it? Expected depth: should describe monitoring the faithfulness guardrail’s fallback rate and downstream regeneration rate as leading indicators, alongside periodic golden-set re-evaluation against current traffic composition, since a golden set built on stale traffic can miss a regression that only shows up in a content type it under-represents.

Q: A serving layer downstream reports its prefix cache hit rate dropped after this pipeline was deployed. What would you check? Expected depth: should identify non-deterministic tie-breaking in the pruning engine or scorer as the likely cause, since identical dynamic content producing different compressed output on different requests breaks the assumption prefix caching depends on. Should also verify the stable prefix itself was never routed through the compression pipeline at all.

Q: If your cost budget got cut in half tomorrow, how would you adjust this system, and what would you expect to break first? Expected depth: should reason about tightening the target compression ratio via the policy engine rather than removing the guardrail, and should predict that downstream accuracy degrades before cost savings plateau, since a harder ratio increases the odds of dropping load-bearing content and increases guardrail fallback rate, which partially offsets the savings.