A 70B parameter model running on 4xA100 GPUs produces roughly 18-22 tokens per second per sequence. That works fine for batch offline processing - but for a real-time chat product, it translates to first tokens arriving 800ms after the user hits send, and visible letter-by-letter streaming that feels like watching paint dry. Users tolerate 200ms. At 800ms they bounce.

The fundamental constraint is that autoregressive decoding forces the model to execute a full forward pass - touching all 140 GB of weights across all 80 transformer layers - for every single token. A 200-token response requires 200 sequential forward passes. There is no way around this with standard decoding; parallelism at the batch level helps throughput but does nothing for per-sequence latency.

Speculative decoding reframes the problem. Think of it like a senior engineer reviewing a junior engineer’s code. The junior (a cheap 8B model) writes a plausible draft quickly. The senior (the 70B model) reviews the whole draft in a single focused pass - approving most of it, correcting the odd token, and adding one clean next step. The output is mathematically equivalent to what the senior would have produced alone, but the wall-clock time is dominated by the single review pass, not by writing each line from scratch.

At 80% acceptance rate with speculation depth gamma=4, a single 70B forward pass produces 3.5 tokens on average instead of 1. With gamma=6 and a high-quality draft model, we can push that to 4.8 tokens per pass. On Llama-3 70B, this translates from 20 tokens per second to 70-80 tokens per second, cutting TTFT from 800ms to under 150ms and TPOT (time per output token) from 50ms to 14ms.

Requirements and Constraints

Functional requirements:

• Accept a prompt and stream back generated tokens with identical statistical distribution to standard autoregressive decoding from the target model
• Support configurable speculation depth gamma (2-8 tokens)
• Handle rejection of draft tokens gracefully via rollback without visible output artifacts
• Support tree-based multi-candidate drafting (Medusa-style) in addition to linear drafting
• Dynamically adjust gamma per sequence based on observed acceptance rate

Non-functional requirements:

• p50 TTFT under 100ms, p99 TTFT under 150ms for prompts up to 512 tokens
• p50 TPOT under 18ms (target: 14ms with effective gamma around 4.5)
• Draft model memory overhead below 20% of target model HBM footprint
• Mathematically lossless: output distribution identical to sampling from target model alone
• Zero added latency when draft acceptance rate falls below threshold - fall back to standard decoding

Constraints:

• Draft model must share the same vocabulary and tokenizer as the target model (otherwise token IDs do not align for verification)
• Draft and target models must be co-located on the same GPU node to avoid PCIe transfer overhead on every speculation step
• Total GPU HBM budget: 80GB per A100, 4 GPUs per node (320 GB total); target model needs 140 GB, leaving 180 GB for KV caches and the draft model

High-Level Architecture

The system has five moving parts. The Speculation Scheduler manages incoming requests and maintains a live gamma configuration per active sequence. The Draft Model (Llama-3 8B) runs the fast autoregressive generation, producing gamma draft tokens with their probability distributions. The Speculation Engine takes those draft tokens plus the target model’s verification pass and runs the accept/reject loop. The Target Model (Llama-3 70B) runs a single forward pass over the entire input including draft tokens. The KV Cache Manager maintains separate caches for draft and target models, handling rollback atomically when tokens are rejected.

Component Deep Dives

The Accept/Reject Mechanism

The non-obvious property of speculative decoding is that it is mathematically lossless - the output distribution is identical to the target model sampling alone. This is not an approximation. It is guaranteed by the rejection sampling correction.

For each draft token d_i, the draft model produced it with probability q(d_i | x) and the target model assigns it probability p(d_i | x). The acceptance rule is: accept d_i with probability min(1, p(d_i|x) / q(d_i|x)). If rejected, resample from the residual distribution normalize(max(0, p(t|x) - q(t|x))) for all tokens t in the vocabulary.

This rule has two critical properties. First, if the draft model proposed d_i and the target agrees it’s at least as likely as the draft thought, it is always accepted (ratio >= 1). Second, if the target rejects, the correction token is sampled from exactly the distribution the target would have produced had it been sampling unconditionally. The composition of these two cases provably recovers the target’s marginal distribution at every position.

After any rejection, all subsequent draft tokens are discarded and the KV cache is rolled back to the position of the last accepted token. This is why rollback handling must be atomic - if the KV cache for the target gets out of sync with the accepted token position, subsequent forward passes will attend over a stale context and produce incorrect outputs.

import torch
from dataclasses import dataclass
from typing import Optional

@dataclass
class SpeculationResult:
    accepted_tokens: list[int]
    correction_token: Optional[int]
    n_accepted: int
    n_rejected: int
    rollback_to_position: int

def speculative_accept_reject(
    draft_token_ids: list[int],
    draft_logits: torch.Tensor,   # shape: [gamma, vocab_size]
    target_logits: torch.Tensor,  # shape: [gamma+1, vocab_size] - one extra for correction
    temperature: float = 1.0,
) -> SpeculationResult:
    """
    Run the speculative decoding accept/reject loop.
    Returns accepted tokens and an optional correction token.
    Output distribution is identical to sampling from target alone.
    """
    gamma = len(draft_token_ids)
    vocab_size = draft_logits.shape[-1]

    # Convert logits to probabilities
    if temperature != 1.0:
        draft_probs = torch.softmax(draft_logits / temperature, dim=-1)
        target_probs = torch.softmax(target_logits / temperature, dim=-1)
    else:
        draft_probs = torch.softmax(draft_logits, dim=-1)
        target_probs = torch.softmax(target_logits, dim=-1)

    accepted = []
    for i, draft_tok in enumerate(draft_token_ids):
        p = target_probs[i, draft_tok].item()
        q = draft_probs[i, draft_tok].item()
        # Clamp to avoid division-by-zero on zero-probability draft tokens
        q = max(q, 1e-10)
        accept_prob = min(1.0, p / q)

        if torch.rand(1).item() < accept_prob:
            accepted.append(draft_tok)
        else:
            # Rejection: sample correction from residual distribution
            residual = torch.clamp(target_probs[i] - draft_probs[i], min=0.0)
            residual_sum = residual.sum()
            if residual_sum > 1e-8:
                residual = residual / residual_sum
                correction = torch.multinomial(residual, num_samples=1).item()
            else:
                # Fallback: sample from target directly (numerical edge case)
                correction = torch.multinomial(target_probs[i], num_samples=1).item()
            return SpeculationResult(
                accepted_tokens=accepted,
                correction_token=correction,
                n_accepted=len(accepted),
                n_rejected=gamma - len(accepted),
                rollback_to_position=len(accepted),
            )

    # All draft tokens accepted - take one more free token from target's (gamma+1)-th position
    bonus_token = torch.multinomial(target_probs[gamma], num_samples=1).item()
    return SpeculationResult(
        accepted_tokens=accepted,
        correction_token=bonus_token,
        n_accepted=gamma,
        n_rejected=0,
        rollback_to_position=gamma,
    )

Draft Model Selection and Alignment

The draft model constraint is stricter than it first appears. The draft and target must share an identical vocabulary and tokenizer - not just the same vocabulary size. A draft model with 128,256 tokens in a different order produces token IDs that point to different words than the target expects, making the acceptance probability ratio p/q meaningless.

In practice, this means the draft model must come from the same model family as the target. Llama-3 8B works as a draft for Llama-3 70B because they share the tiktoken-based tokenizer exactly. Mistral 7B does not work as a draft for Llama-3 70B even though both have 128K vocabulary, because their tokenization of the same string produces different token ID sequences.

Draft model candidates for Llama-3 70B in production order of preference:

Draft Model	Parameter Ratio	HBM (BF16)	Typical Acceptance Rate	Notes
Llama-3 8B	8.6x	16 GB	80-85%	Best choice; same family, highest correlation
Llama-3.1 8B	8.6x	16 GB	82-87%	Slightly better on code and structured text
Llama-3 1B (pruned)	70x	2 GB	65-72%	Marginal; fast but low acceptance hurts throughput
Medusa heads on 70B	N/A	+2 GB	72-78%	Fine-tuned heads, no separate model needed

The parameter ratio roughly determines acceptance rate. Models from the same training lineage but with fewer parameters will have learned similar token predictions for common continuations, which is precisely when speculation helps most. High-entropy continuations (creative writing, code generation past function signatures) see lower acceptance rates regardless of model choice.

KV Cache Management for Dual Models

Running two models on the same GPU node means managing two distinct KV caches that must stay in lockstep during speculation and diverge during rollback. The non-obvious constraint is that both caches must be updated together during accepted tokens and both must be rolled back together during rejections.

class DualKVCacheManager:
    """
    Manages KV caches for both draft and target models.
    Rollback must be atomic - both caches must reflect the same sequence length.
    """
    def __init__(
        self,
        draft_num_layers: int,   # 32 for Llama-3 8B
        target_num_layers: int,  # 80 for Llama-3 70B
        num_kv_heads_draft: int, # 8 (GQA)
        num_kv_heads_target: int,# 8 (GQA)
        head_dim: int,           # 128
        max_seq_len: int,        # 8192
        dtype: torch.dtype = torch.bfloat16,
    ):
        self.accepted_len = 0
        # Draft KV cache: [batch, 2, layers, seq, heads, head_dim]
        self.draft_kv = torch.zeros(
            2, draft_num_layers, max_seq_len, num_kv_heads_draft, head_dim,
            dtype=dtype, device='cuda'
        )
        # Target KV cache: same structure but target dimensions
        self.target_kv = torch.zeros(
            2, target_num_layers, max_seq_len, num_kv_heads_target, head_dim,
            dtype=dtype, device='cuda'
        )

    def commit_speculation(self, n_accepted: int):
        """
        After accept/reject loop, commit n_accepted tokens.
        Invalidates the phantom tail in target cache.
        """
        self.accepted_len += n_accepted
        # The target cache has entries up to accepted_len + gamma; mark the tail invalid
        # by tracking the "valid_len" pointer - no memset needed, just update the pointer
        # and pass it as kv_cache_len in the next forward pass
        return self.accepted_len

    def rollback(self, target_len: int):
        """
        Roll back both caches to target_len.
        This is a pointer update only - the memory is reused on the next forward pass.
        """
        self.accepted_len = target_len
        return self.accepted_len

    def get_draft_cache_slice(self) -> torch.Tensor:
        """Return the valid portion of the draft KV cache."""
        return self.draft_kv[:, :, :self.accepted_len, :, :]

    def get_target_cache_slice(self) -> torch.Tensor:
        """Return the valid portion of the target KV cache."""
        return self.target_kv[:, :, :self.accepted_len, :, :]

The rollback is a pointer update, not a memory clear. Clearing GPU memory is expensive; simply not reading past valid_len achieves the same effect because the next forward pass overwrites those positions anyway.

Speculation Depth Tuning and Tree Drafting

Gamma (speculation depth) is the single most impactful tunable parameter. Setting it too low wastes the potential speedup from high acceptance rates. Setting it too high means the draft model runs longer than necessary on low-acceptance sequences, and the target’s context becomes padded with tokens that will all be rejected.

The expected number of accepted tokens per target forward pass follows a geometric distribution. If each draft token is accepted independently with probability alpha, then the expected accepted count is:

E[accepted] = sum_{k=0}^{gamma} alpha^k * (k if k==gamma else k)
            = (1 - alpha^(gamma+1)) / (1 - alpha)

At alpha=0.80, gamma=4: E[accepted] = (1 - 0.8^5) / 0.2 = (1 - 0.328) / 0.2 = 3.36
At alpha=0.80, gamma=8: E[accepted] = (1 - 0.8^9) / 0.2 = (1 - 0.134) / 0.2 = 4.33
At alpha=0.65, gamma=4: E[accepted] = (1 - 0.65^5) / 0.35 = 2.36
At alpha=0.65, gamma=8: E[accepted] = (1 - 0.65^9) / 0.35 = 2.72

The marginal gain from increasing gamma shrinks quickly at lower acceptance rates. For alpha below 0.65, increasing gamma beyond 4 adds draft model latency without proportional acceptance gain. The Gamma Controller monitors the rolling acceptance rate per sequence and adjusts gamma dynamically.

Tree speculation (Medusa-style) sidesteps the gamma tuning problem by generating a tree of candidate continuations instead of a single linear chain. Medusa adds dedicated prediction heads on top of the target model’s hidden states - typically 4-5 heads, each predicting a different future token position independently. These heads are small FFN layers (a few million parameters) trained with a separate supervised objective. During inference, head i predicts the token at position +i, producing top-k candidates per head. The cartesian product of these candidates forms the speculation tree.

The target model then runs its forward pass over the entire tree in one shot using a custom tree attention mask that only lets each candidate attend over its own ancestor chain. This is more complex to implement than linear speculation but delivers higher expected acceptance because the target can accept different branches of the tree for different sequence positions.

def build_medusa_tree_mask(tree_candidates: list[list[int]]) -> torch.Tensor:
    """
    Build attention mask for tree-based speculative decoding.
    Each candidate token can only attend to its ancestors in the tree.
    tree_candidates: list of token sequences (branches), each starting from root.
    Returns: [total_candidates, total_candidates] bool mask
    """
    # Flatten tree into a list of nodes with parent pointers
    nodes = []  # (token_id, parent_idx)
    nodes.append((-1, -1))  # root (last accepted token, attends to all prompt)

    for branch in tree_candidates:
        parent_idx = 0  # all branches start from root
        for depth, tok in enumerate(branch):
            nodes.append((tok, parent_idx))
            parent_idx = len(nodes) - 1

    n = len(nodes)
    mask = torch.zeros(n, n, dtype=torch.bool)

    def ancestors(idx: int) -> set:
        result = set()
        while idx != -1:
            result.add(idx)
            idx = nodes[idx][1]
        return result

    for i in range(n):
        ancs = ancestors(i)
        for j in range(n):
            if j in ancs:
                mask[i, j] = True

    return mask

TTFT and TPOT: What Speculation Actually Improves

Time-to-first-token (TTFT) measures how long the user waits before seeing any output. Time-per-output-token (TPOT) measures how fast tokens stream once they start. These metrics decompose independently and speculation improves them differently.

TTFT is dominated by the prefill phase (processing the input prompt). A 512-token prompt on Llama-3 70B takes about 140ms for prefill - the entire prompt is processed in one batched matrix multiply, which is compute-bound and fast. Speculation does not help prefill because prefill is already parallelized across all prompt tokens.

TPOT is where speculation shines. Standard decoding on 70B: 18ms per forward pass, 1 token output. With speculative decoding at 3.5 tokens per pass: 10ms draft time + 20ms target pass = 30ms for 3.5 tokens = 8.6ms per token. At a streaming rate of 8.6ms/token, the user sees text appearing at 116 tokens per second - indistinguishable from real-time for reading speed.

The total generation time for a 200-token response:

• Standard: prefill 140ms + 200 * 18ms decode = 3,740ms
• Speculative (alpha=0.80, gamma=4): prefill 140ms + (200/3.5) * 30ms = 140ms + 1,714ms = 1,854ms
• Effective speedup: 2.0x on total wall-clock, 3.5x on decode phase alone

The headline “3x speedup” refers to the decode phase. Total generation time including prefill is around 2x because prefill is unchanged. For short prompts and long outputs (the common chat case), decode dominates and the 3x figure is accurate.

Data Model

-- Speculation state per active sequence
CREATE TABLE speculation_sessions (
    session_id          UUID PRIMARY KEY,
    request_id          UUID NOT NULL,
    created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    prompt_token_count  INTEGER NOT NULL,
    generated_tokens    INTEGER NOT NULL DEFAULT 0,
    current_gamma       SMALLINT NOT NULL DEFAULT 4,
    -- Rolling acceptance statistics (last 20 steps)
    rolling_accept_sum  SMALLINT NOT NULL DEFAULT 0,
    rolling_accept_n    SMALLINT NOT NULL DEFAULT 0,
    total_accepted      INTEGER NOT NULL DEFAULT 0,
    total_rejected      INTEGER NOT NULL DEFAULT 0,
    total_target_passes INTEGER NOT NULL DEFAULT 0,
    -- KV cache pointers
    draft_kv_ptr        BIGINT,    -- GPU memory address of draft KV slice
    target_kv_ptr       BIGINT,    -- GPU memory address of target KV slice
    kv_valid_len        INTEGER NOT NULL DEFAULT 0,
    -- Latency tracking
    last_ttft_ms        FLOAT,
    last_tpot_ms        FLOAT,
    model_version       VARCHAR(64) NOT NULL,
    status              VARCHAR(16) NOT NULL DEFAULT 'active'  -- active|completed|failed
);

-- Per-step telemetry (written async, not on hot path)
CREATE TABLE speculation_steps (
    step_id         BIGSERIAL PRIMARY KEY,
    session_id      UUID NOT NULL REFERENCES speculation_sessions(session_id),
    step_num        INTEGER NOT NULL,
    gamma_used      SMALLINT NOT NULL,
    n_accepted      SMALLINT NOT NULL,
    n_rejected      SMALLINT NOT NULL,
    draft_latency_ms FLOAT NOT NULL,
    target_latency_ms FLOAT NOT NULL,
    effective_tpot_ms FLOAT NOT NULL,
    recorded_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Gamma configuration policy (per model, per content category)
CREATE TABLE gamma_policy (
    policy_id       SERIAL PRIMARY KEY,
    model_id        VARCHAR(64) NOT NULL,
    content_type    VARCHAR(32),  -- 'chat', 'code', 'structured', NULL=default
    gamma_min       SMALLINT NOT NULL DEFAULT 2,
    gamma_max       SMALLINT NOT NULL DEFAULT 8,
    gamma_default   SMALLINT NOT NULL DEFAULT 4,
    alpha_threshold FLOAT NOT NULL DEFAULT 0.65,  -- below this, reduce gamma
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Key Algorithms and Protocols

Dynamic Gamma Controller

The gamma controller watches the rolling acceptance rate and adjusts gamma to maximize tokens per second, not acceptance count. The optimal gamma is not the highest one - it is the one where the marginal gain from another draft token exceeds the marginal draft latency cost.

from collections import deque
import math

class GammaController:
    """
    Adaptive gamma tuner. Tracks rolling acceptance rate and adjusts
    speculation depth to maximize effective tokens per second.
    """
    WINDOW = 20  # steps to average over

    def __init__(self, gamma_min: int = 2, gamma_max: int = 8, gamma_init: int = 4):
        self.gamma = gamma_init
        self.gamma_min = gamma_min
        self.gamma_max = gamma_max
        self._window: deque[tuple[int, int]] = deque(maxlen=self.WINDOW)
        # Measured latencies (ms)
        self.draft_ms_per_token = 2.0   # calibrated at startup
        self.target_ms_per_pass = 20.0  # calibrated at startup

    def record_step(self, n_accepted: int, n_rejected: int):
        self._window.append((n_accepted, n_rejected))

    def rolling_alpha(self) -> float:
        if not self._window:
            return 0.80  # default assumption
        total_accepted = sum(a for a, _ in self._window)
        total_steps = sum(a + r for a, r in self._window)
        return total_accepted / max(total_steps, 1)

    def optimal_gamma(self, alpha: float) -> int:
        """
        Find gamma that maximizes (E[accepted] / total_time).
        total_time = gamma * draft_ms_per_token + target_ms_per_pass
        """
        best_gamma = self.gamma_min
        best_toks_per_sec = 0.0
        for g in range(self.gamma_min, self.gamma_max + 1):
            # E[accepted] = (1 - alpha^(g+1)) / (1 - alpha), capped at g
            if alpha >= 1.0:
                e_accepted = float(g)
            else:
                e_accepted = (1 - alpha ** (g + 1)) / (1 - alpha)
            total_ms = g * self.draft_ms_per_token + self.target_ms_per_pass
            toks_per_sec = e_accepted / (total_ms / 1000.0)
            if toks_per_sec > best_toks_per_sec:
                best_toks_per_sec = toks_per_sec
                best_gamma = g
        return best_gamma

    def step(self, n_accepted: int, n_rejected: int) -> int:
        self.record_step(n_accepted, n_rejected)
        if len(self._window) >= 5:  # enough data to make a decision
            alpha = self.rolling_alpha()
            self.gamma = self.optimal_gamma(alpha)
        return self.gamma

Rollback Protocol

Rollback is where implementations diverge. When draft token d_i is rejected, we must:

1. Discard d_i and all subsequent draft tokens
2. Roll back the target KV cache to position N + (i-1) where N is the prompt length
3. Roll back the draft KV cache to the same position
4. Emit the correction token
5. Resume draft generation from the new position

The critical constraint is that steps 2 and 3 must be atomic from the scheduler’s perspective. If the target KV cache is rolled back but the draft KV cache is not (or vice versa), the next speculative step will prefix-extend the wrong cached context. This produces subtly wrong outputs - not a crash, not obviously wrong, just quietly incorrect attention patterns.

In vLLM’s implementation, this is handled by using a paged KV cache with a block pointer table. Rollback means updating the block pointer table for both draft and target in a single synchronized operation before releasing any new compute work to the GPU.

Batch-Level Speculation

Running speculative decoding over a batch of sequences introduces the ragged accept problem. Sequence A might accept all 4 draft tokens while Sequence B rejects at token 2. After the accept/reject loop, sequences in the batch have different accepted lengths - some at position N+4, some at N+2, some at N+1. The next target forward pass must handle all of these simultaneously with a batched input matrix where each row has a different valid prefix length.

vLLM solves this by padding to max_accepted_len + 1 in the batch and using the kv_cache_len metadata per sequence to mask the attention correctly. TensorRT-LLM takes a different approach: it bins sequences by accepted count into micro-batches and processes them separately, avoiding the padding waste at the cost of more GPU kernel launches.

The padding waste at gamma=4 with mixed acceptance is bounded. In the worst case (all sequences reject at different positions), you process a 4-column batch where on average 2 columns are valid. The wasted compute is at most 50% of the target forward pass - still far better than the 4x savings from speculation.

Scaling and Performance

The memory layout for a 4xA100 80GB node running Llama-3 70B + Llama-3 8B speculative decoding:

Hardware: 4x A100 80GB, NVLink 3rd gen (600 GB/s bidirectional)
Total HBM: 320 GB

Model weights (BF16):
  Target Llama-3 70B: 70B * 2 bytes = 140 GB, sharded across 4 GPUs = 35 GB/GPU
  Draft Llama-3 8B:   8B * 2 bytes  = 16 GB, sharded across 2 GPUs = 8 GB/GPU
    (draft only needs 2 GPUs; remaining 2 GPUs used for target overflow + KV)

KV Cache budget:
  Remaining HBM (target GPUs): 80 - 35 = 45 GB each, 2 target-only GPUs at 80 GB each
  Total KV budget: 45*2 + (80-8)*2 = 90 + 144 = 234 GB

KV cache per token (target, GQA with 8 KV heads, 128 dim, 80 layers, BF16):
  2 heads/GPU * 128 * 2 bytes * 80 layers * 2 (K+V) = 81,920 bytes = 80 KB/token/GPU
  Over 4 GPUs total: 320 KB per token for full parallel KV

KV cache per token (draft, 8 KV heads, 128 dim, 32 layers, BF16):
  4 heads/GPU * 128 * 2 bytes * 32 layers * 2 (K+V) = 65,536 bytes = 64 KB/token/GPU

For 8K context window (8,192 tokens):
  Target KV: 8192 * 320 KB = 2.56 GB per sequence
  Draft KV:  8192 * 64 KB  = 512 MB per sequence

Maximum concurrent sequences at 8K context:
  Target KV budget: 234 GB / 2.56 GB = 91 sequences
  Draft KV budget:  (16 GB free on draft node) / 512 MB = 32 sequences
  Bottleneck: draft KV at 32 sequences

At average 2K context (typical chat turn):
  Target KV: 2048 * 320 KB = 640 MB per sequence
  Draft KV:  2048 * 64 KB  = 128 MB per sequence
  Target: 234 GB / 640 MB = 365 sequences
  Draft: 16 GB / 128 MB  = 128 sequences
  Bottleneck: draft KV at 128 concurrent sequences

Throughput at 128 sequences, 3.5 tokens per pass, 30ms per speculation step:
  128 sequences * 3.5 tokens / 0.030 seconds = 14,933 tokens/second
  vs standard decoding: 128 * 1 token / 0.018s = 7,111 tokens/second
  Effective throughput gain: 2.1x (batch efficiency overhead reduces from single-sequence 3.5x)

At 128 concurrent sequences, the draft model’s KV cache is the binding constraint, not the target model’s KV cache. Reducing draft model KV by using a smaller draft model (1B instead of 8B) frees up draft KV but reduces acceptance rate - a direct throughput tradeoff.

Cost and Token Economics

Configuration	Tokens/sec (4xA100)	GPU cost/1M tokens	TPOT p50	TTFT p99
Standard decoding, bs=1	56	$39.68	18ms	165ms
Standard decoding, bs=32	2,275	$0.97	18ms	230ms
Speculative (gamma=4, alpha=0.80) bs=32	5,040	$0.44	8.3ms	155ms
Speculative (gamma=6, alpha=0.80) bs=32	5,810	$0.38	7.2ms	160ms
Medusa (4 heads, top-3 tree) bs=32	6,200	$0.36	6.7ms	150ms

The most significant observation in this table: batch size 32 without speculation already cuts cost by 40x compared to batch size 1. Speculative decoding then provides an additional 2.2x on top of that. The absolute best cost position is Medusa with tree drafting at batch size 32 - $0.36 per 1M output tokens versus $39.68 for naive single-sequence serving.

The crossover point where speculation stops helping is around alpha < 0.55. Below that, draft model latency costs more than the acceptance gains pay for. This happens in practice on very high-entropy generation tasks: creative fiction with wide token distributions, code generation at the end of function bodies, and multi-language mixed outputs where the draft model’s training distribution diverges from the target’s.

Quality, Evaluation, and Guardrails

Speculative decoding is mathematically lossless by construction, but three implementation bugs can break this guarantee silently:

1. Temperature inconsistency. The acceptance probability min(1, p/q) is only valid when both p and q are computed at the same temperature. If the draft model applies temperature 0.8 and the target applies temperature 1.0, the ratio is wrong and the output distribution shifts. Always apply temperature before computing probabilities, and apply the same temperature to both models.

2. KV cache misalignment. If the target KV cache is not rolled back to exactly the last accepted position before the next forward pass, the target attends over phantom tokens. The output looks plausible but is conditioned on a different history than the user sees. This is the hardest bug to detect because the outputs are still coherent text.

3. Sampling seed propagation. Reproducible generation (for debugging, A/B testing) requires that the acceptance coin flips and correction token samples use a deterministic RNG seeded from the request. A non-deterministic RNG produces non-reproducible outputs even when the model weights and prompt are identical.

# Eval harness: verify output distribution equivalence between speculative and standard decoding
import numpy as np
from scipy.stats import ks_2samp

def verify_distribution_equivalence(
    standard_outputs: list[list[int]],   # N runs with standard decoding
    speculative_outputs: list[list[int]], # N runs with speculative decoding
    n_positions: int = 50,  # check first 50 token positions
) -> dict:
    """
    KS test to verify speculative decoding matches target distribution.
    Run 1000 samples at temperature > 0 to get a distribution.
    Returns: per-position p-values (should all be > 0.05)
    """
    results = {}
    for pos in range(n_positions):
        std_toks = [seq[pos] for seq in standard_outputs if len(seq) > pos]
        spec_toks = [seq[pos] for seq in speculative_outputs if len(seq) > pos]
        if len(std_toks) < 30 or len(spec_toks) < 30:
            continue
        stat, p_value = ks_2samp(std_toks, spec_toks)
        results[pos] = {'ks_stat': stat, 'p_value': p_value, 'pass': p_value > 0.05}
    overall_pass = all(r['pass'] for r in results.values())
    return {'per_position': results, 'overall_pass': overall_pass,
            'fail_count': sum(1 for r in results.values() if not r['pass'])}

The KS test across 1000 samples at each token position is the gold standard for validating speculative decoding correctness before shipping. A p-value below 0.05 at any position is a signal of an implementation bug. Common culprits: floating-point precision differences between draft and target softmax implementations, and off-by-one errors in KV cache indexing.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Draft model OOM on GPU	Draft forward pass CUDA OOM exception	Speculation unavailable on affected sequences	Fall back to standard decoding on affected sequences; alert for draft KV cache eviction
KV cache desync (target and draft at different lengths)	target_kv_len != draft_kv_len assertion	Silent quality degradation - target attends over wrong context	Hard reset both KV caches for the session; restart generation from prompt prefill
Acceptance rate collapse (alpha < 0.40)	Rolling alpha monitor; alert if alpha < 0.45 for 10 consecutive steps	Speculation overhead exceeds benefit; latency worse than standard	Disable speculation for the sequence; set gamma=0 until alpha recovers
Draft model version mismatch after deploy	Model version field check on session init	Draft and target vocabularies diverge; token IDs misaligned	Block session creation until draft and target share the same model family version; rolling deploy requires both models to update together
Batch padding explosion (max gamma outlier)	Batch padded_len / avg_accepted_len ratio > 3x	Wasted GPU compute on padding; throughput regression	Cap gamma at p95 of observed accepted lengths; remove outlier sequences from batch into separate micro-batch
Temperature mismatch between draft and target	Distribution test fails in shadow mode	Output distribution shifts; subtly biased generation	Enforce shared temperature config object; reject inference requests that specify per-model temperatures independently
Rollback race condition (async KV flush)	Non-deterministic test failures; occasional repeated tokens	One token output twice or skipped	Serialize rollback and next-step scheduling; no async KV update allowed between rollback and next draft step

Comparison of Approaches

Approach	Throughput Gain	TPOT Improvement	Memory Overhead	Implementation Complexity	Best Fit
Standard autoregressive	1x (baseline)	0ms improvement	None	Minimal	Low-latency requirements not needed; max batch size priority
Linear speculative decoding (separate draft model)	2.5-3.5x	60-70% reduction	+5-20% HBM (draft model)	Medium: dual KV cache, rollback logic	Production chat with same-family 8B draft available
Medusa (auxiliary prediction heads)	2.2-3.6x	55-65% reduction	+1-3% HBM (head weights only)	High: head training, tree attention mask	When no separate draft model is available; fine-tuning budget exists
EAGLE (draft from target’s feature map)	3.0-4.0x	65-75% reduction	+2-4% HBM	Very high: requires target feature extraction pipeline	Highest-quality draft correlation; specialized engineering capacity
Lookahead decoding (n-gram draft)	1.5-2.0x	30-45% reduction	Minimal (n-gram table in CPU)	Low: no draft model needed	When GPU memory budget prohibits any model co-location
Parallel sampling then select	1.0x (same latency, more diverse outputs)	0	N x model copies	Low	Diversity over latency; batch diversity applications

Linear speculative decoding with a same-family draft model is the right default for production chat. The implementation is well-understood, vLLM and TensorRT-LLM both ship it production-ready, and the 2.5-3.5x gain is consistent across chat workloads. Medusa becomes the right choice when you cannot afford the HBM for a separate draft model (already near the limit with the target) and have a fine-tuning pipeline to train the heads. EAGLE is the frontier approach - it uses the target model’s own intermediate layer outputs as the draft signal, achieving the highest acceptance rates but requiring invasive changes to the target’s forward pass.

Key Takeaways

• Speculation is lossless by construction - the rejection sampling correction recovers the exact target distribution at every position; it is not an approximation and cannot degrade output quality when implemented correctly.
• Draft model family matching is mandatory - the draft must share the exact tokenizer and vocabulary with the target, which in practice means same model family (e.g., Llama-3 8B drafting for Llama-3 70B); cross-family drafting produces wrong acceptance ratios.
• Gamma tuning is critical and dynamic - the optimal gamma depends on the rolling acceptance rate of the current sequence; a static gamma=4 is acceptable as a default but a dynamic gamma controller extracts another 15-20% throughput.
• KV cache rollback must be atomic - both draft and target KV caches must reflect identical sequence lengths at all times; any desync produces silent quality degradation that is hard to detect without explicit length assertions.
• Batch speculation introduces padding overhead - ragged acceptance lengths across a batch require padding to max accepted length, eating into per-sequence gains; at batch size 32, expect 2.0-2.5x gain rather than the single-sequence 3.5x.
• All gamma tokens accepted is better than partial acceptance - when all draft tokens accept, the target’s (gamma+1)-th output is a free bonus token; the expected tokens per pass is (1 - alpha^(gamma+1)) / (1 - alpha), which exceeds gamma at alpha < 1.
• Medusa tree drafting raises the ceiling - by generating a tree of candidates instead of a linear chain, Medusa extracts more accepted tokens per target pass at the same acceptance probability; the tradeoff is tree attention mask complexity and the need to train the prediction heads.
• Fallback to standard decoding must be seamless - sequences with collapsed acceptance rates (very high-entropy domains like creative poetry, code generation at decision branches) should drop out of speculation without visible output gaps or latency spikes.

Frequently Asked Questions

Q: Does speculative decoding change the output of the model? Will users get different responses?

A: No, when implemented correctly. The accept/reject mechanism with the residual resampling correction guarantees that the output distribution is mathematically identical to sampling from the target model alone. The only way outputs can change is if there is an implementation bug (temperature mismatch, KV cache desync, incorrect residual computation). The statistical equivalence can be verified with a KS test across a few thousand samples per token position.

Q: Why does speculative decoding help at all if the target model still runs a forward pass every step?

A: The key is that the target’s forward pass is over gamma+1 positions simultaneously, and the cost of a forward pass over N+gamma tokens is only marginally more expensive than over N tokens (the attention cost is quadratic in sequence length, but the FFN cost and most of the memory bandwidth cost is linear in batch size). The single target pass that verifies 4 draft tokens costs roughly the same as a standard 1-token decode step. You are getting 2.5-4 accepted tokens for the same wall-clock cost as 1.

Q: What happens to speculative decoding when you use beam search instead of sampling?

A: Classic speculative decoding is designed for sampling (temperature > 0). Greedy decoding (temperature = 0) simplifies the acceptance rule: accept if and only if the draft token equals the target’s argmax. This gives a clean accept/reject but typically lower acceptance rates because even small distribution differences cause greedy tokens to diverge. For beam search, speculation does not directly apply - each beam has an independent decode state and verifying all beams simultaneously would require running the target over B * (gamma+1) positions, which only helps if B is small and acceptance is very high.

Q: How do you handle speculative decoding with constrained generation (JSON mode, grammar constraints)?

A: Constrained generation adds a rejection mask on top of the normal token distribution. Both draft and target must apply the same constraint mask before computing probabilities. If the draft model applies the constraint but the target does not (or applies a different constraint state), the acceptance ratios are wrong. The correct implementation runs the constraint automaton state machine in lockstep for both models, always masking to the same allowed token set. vLLM’s constrained speculative decoding does this via the logits_processor interface applied uniformly to both models.

Q: Can speculative decoding help with the prefill phase, not just decode?

A: Standard speculation does not help prefill because prefill already processes all tokens in parallel. However, “speculative prefill” is an active research area: the idea is to use the draft model to predict chunks of the prompt that can be skipped or approximated in the target’s attention computation. Distrifusion and similar methods do this for diffusion models. For autoregressive LLMs, prompt caching (KV cache sharing for common prefixes) is the more practical optimization for reducing repeat-prefill cost - speculation addresses the decode bottleneck.

Q: How does vLLM implement speculative decoding in practice?

A: vLLM’s speculative decoding (added in v0.3.0) uses a draft model worker alongside the target model worker, both sharing the same paged KV cache block allocator. The scheduler runs draft generation for the entire batch synchronously before running the target verification pass. Rollback is a page table update - pages corresponding to rejected positions are returned to the free list rather than being zeroed. The gamma is configurable per request via SamplingParams.speculative_tokens. As of vLLM 0.5, Medusa heads are supported as an alternative draft mechanism using the same scheduler and rollback infrastructure.

Interview Questions

Q: Walk through the mathematical proof that speculative decoding is lossless. What property of the residual distribution makes it work?

Expected depth: The proof hinges on the law of total probability. For any token t, the probability of outputting t under speculation equals: P(draft proposes t) * P(accept) + P(draft proposes t’) * P(reject t’) * P(resample t from residual). Expanding: q(t) * min(1, p(t)/q(t)) + sum_{t' != t} q(t') * (1 - min(1, p(t')/q(t'))) * residual(t). After substituting the residual definition normalize(max(0, p - q)) and doing the algebra, this collapses to exactly p(t). The key insight: the normalization of max(0, p-q) divides by sum_t max(0, p(t)-q(t)) which equals sum_t (p(t)-q(t))^+ = the total variation distance where p exceeds q. The expected depth hint should also mention the bonus token case (all accepted) and why it doesn’t break the marginal.

Q: You’re running Llama-3 70B with a Llama-3 8B draft on 4xA100 80GB. At what batch size does speculative decoding stop improving TPOT, and why?

Expected depth: Walk through: at small batch sizes (1-8), the GPU is underutilized during the target forward pass; both draft and target are memory-bandwidth-bound, not compute-bound. Adding speculation increases tokens per pass without changing the memory bandwidth bottleneck. As batch size grows to 32-64, the target forward pass becomes compute-bound (matrix multiplies hit peak FLOPS). At this point, the (gamma+1)-wide target pass is more expensive than a 1-wide pass because the additional columns in the activation matrix consume more FLOPS. The crossover is around batch size 64-128 depending on hardware. Above that, speculation may slightly reduce TPOT improvement because the target forward pass cost grows superlinearly with speculation width. The candidate should know that for throughput-optimized serving (large batches), speculation helps less; it is primarily a latency optimization for small-to-medium batch sizes.

Q: A senior engineer proposes using GPT-4o-mini as the draft model for GPT-4o. What’s wrong with this?

Expected depth: The core issue is vocabulary alignment. GPT-4o-mini and GPT-4o share the same tiktoken cl100k_base tokenizer, so token IDs do align - this is actually a valid pairing if both are accessible. However, the real constraint is access: you cannot run GPT-4o’s forward pass to get p(d_i|x) for token-level verification unless you have white-box access to GPT-4o’s logits. API-only access gives you only the sampled output, not the full probability distribution needed for the min(1, p/q) acceptance ratio. This is why speculative decoding is only practical for self-hosted or white-box model deployments. A deeper answer would also note that even with white-box access, the models must run on the same compute cluster (same GPU node, ideally same NVLink domain) to avoid network latency overwhelming the speculation gain.

Q: Your speculative decoding system has 80% acceptance rate in testing but only 61% in production. What are the likely causes and how do you debug each?

Expected depth: Distribution shift between test and production prompts is the most likely cause - test prompts may be shorter, more templated, or from a different domain than production. Debugging steps: log acceptance rate by prompt length, by detected content type (code vs chat vs structured), and by time-of-day (to catch workload shifts). Second cause: temperature mismatch - if production uses a different temperature schedule than testing (e.g., adaptive temperature based on position in the response), the draft-target probability ratio changes. Third: tokenizer version drift - if a tokenizer update shipped to production but not to the test environment, the same string tokenizes to different IDs and the acceptance ratio computation is wrong. Debug by running identical requests through both environments and comparing per-token acceptance decisions.

Q: Design the gamma controller for a multi-tenant serving system where different request types have very different optimal gammas. How do you avoid per-request overhead while keeping gamma adaptive?

Expected depth: Start by noting that a global gamma is suboptimal: code generation might benefit from gamma=6 (structured text with high acceptance) while creative writing needs gamma=2 (high entropy, low acceptance). The solution is to maintain per-content-type gamma estimates using the content_type field from the request header. A secondary model or rule-based classifier assigns each request a content type on arrival (100-microsecond overhead). The gamma controller maintains separate rolling acceptance rate windows per content type and a separate optimal gamma per type. Per-sequence adaptation within a content type happens using the geometric distribution formula. The critical implementation detail is that the gamma decision must be made before allocating KV cache blocks for the draft model - allocating gamma=6 worth of draft KV and then only using 2 tokens is wasteful. The adaptive controller should commit to a gamma after the first 3-4 steps of a new sequence, not before any steps, using the content-type prior as the initial gamma.