← Reference · Nestor G Pestelos Jr · Print this page

Machine Learning · Systems Engineering

Speculative Decoding

Reference entry · last updated September 8, 2026

Speculative decoding (also termed assisted generation or speculative sampling) is an exact acceleration technique for autoregressive language model inference that uses an auxiliary draft mechanism to hypothesize candidate tokens for parallel validation by a larger target model in a single execution step. Because validating multiple tokens concurrently is compute-bound rather than memory-bound, speculative decoding reduces generation latency while provably preserving the target model's output probability distribution.

1. First Principles: Parallel Verification vs. Sequential Generation

Autoregressive text generation requires \(K\) sequential steps to emit \(K\) tokens. Because decoding each token is memory-bandwidth bound, generating \(K\) tokens requires reading the target model's entire parameter weight matrix \(P\) from memory \(K\) times [1, 2]. This serial dependency enforces an architectural latency barrier governed by memory bus throughput.

However, verifying \(K\) candidate tokens simultaneously requires only a single forward pass of the target model. Evaluating a sequence of length \(K\) performs matrix-matrix operations (GEMM) across the sequence dimension rather than matrix-vector operations (GEMV), unlocking high tensor core arithmetic intensity. Verifying \(K\) candidate tokens takes approximately the same wall-clock time on a modern GPU as generating a single token sequentially.

Speculative decoding exploits this asymmetry: a lightweight, fast draft mechanism generates \(K\) candidate tokens at low computational cost, and the large target model inspects all \(K\) candidates in parallel in one forward pass [1].

2. Algorithmic Formulation and Distribution Preservation

2.1 Modified Rejection Sampling

A key property of speculative decoding is mathematical exactness: the generated text distribution matches sampling from the target model \(M_{\text{target}}\) exactly, introducing zero degradation in output quality or reasoning capability [1, 2].

Let \(q(x)\) denote the token probability distribution from the draft model, and \(p(x)\) denote the true distribution from the target model. For each draft token \(x_i\) conditioned on preceding accepted tokens, the verification step accepts \(x_i\) with probability:

If draft token \(x_i\) is accepted, the process evaluates candidate \(x_{i+1}\). If token \(x_i\) is rejected, speculative execution halts for that iteration. The algorithm draws a replacement token from adjusted distribution \(p'(x)\):

This rejection sampling formulation guarantees that whether a token is accepted from the draft or sampled from the residual distribution, the marginal probability of emitting token \(x\) equals \(p(x)\) identically.

2.2 Expected Speedup and Acceptance Rate

If the draft model produces \(\gamma\) candidate tokens per iteration and achieves an average per-token acceptance rate \(\alpha \in [0, 1]\), the expected count of generated tokens \(\mathbb{E}[N]\) per verification pass is:

Let \(c\) represent the latency ratio between one draft step and one target model forward pass (\(c = T_{\text{draft}} / T_{\text{target}} \ll 1\)). The wall-clock speedup \(S\) is expressed as:

When \(\alpha \approx 0.7\) to \(0.9\) and \(c \le 0.05\), speculative decoding yields a 2x to 3x reduction in Time Per Output Token (TPOT).

3. Drafting Architectures

3.1 Auxiliary Draft Models

The standard architecture pairs a small model from the same family with the large target model (for example, a 7B parameter draft model proposing tokens for a 70B parameter target model) [1]. Both models share the same vocabulary and tokenization scheme, allowing straightforward probability mapping.

3.2 Multi-Head and Non-Neural Drafting

Modern serving frameworks also support draft mechanisms that eliminate secondary models:

• Multi-Head Speculation (Medusa): Several lightweight feed-forward heads are trained on top of the target model's final hidden state, predicting subsequent tokens \(t+1, t+2, \dots\) simultaneously without running a secondary model [3].
• Feature-Recurrent Drafting (EAGLE): Extends multi-head prediction by feeding target transformer feature vectors into a single transformer decoder layer, capturing contextual representations with high acceptance accuracy [4].
• Prompt Lookup / N-Gram Matching: Reuses matching token n-grams from the input context without neural execution. Effective in summarization, translation, and code editing where output strings heavily repeat prompt text.

4. Operating Regimes and Systems Trade-offs

Speculative decoding operates most effectively under specific system constraints:

• Low Concurrency Regimes: Speculative decoding provides maximum acceleration at small batch sizes (concurrency = 1 to 4), where the GPU is memory-bandwidth bound. At very large batch sizes, target model execution is already compute-saturated, reducing the efficiency of speculative verification.
• Entropy Dependence: Acceptance rate \(\alpha\) depends on output entropy. Predictable syntax (JSON structures, code boilerplate, factual names) yields \(\alpha > 0.85\), producing dramatic acceleration. High-entropy creative text or dense mathematical reasoning yields lower acceptance rates (\(\alpha < 0.5\)), attenuating speedup.
• Memory Footprint: Running a dual-model setup requires loading both draft and target weights into accelerator memory, slightly reducing available KV cache capacity.