What Is Speculative Decoding? How It Cuts AI Inference Costs by 3-6x

The Autoregressive Bottleneck

Every large language model generates text one token at a time. Each token requires a full forward pass through billions of parameters, and the next token can't start until the previous one finishes. This sequential bottleneck means that a 400B parameter model generating 2,000 tokens keeps a GPU cluster occupied for 20-40 seconds — regardless of whether those tokens are trivially predictable boilerplate or genuinely novel reasoning.

Speculative decoding breaks this bottleneck by separating the "what to generate" step from the "is this correct" step — and making the verification step massively parallel.

How Standard Speculative Decoding Works

The core idea is simple: use a small, fast "draft" model to propose N tokens ahead, then have the large "verifier" model check all N tokens in a single forward pass. Here's the step-by-step:

• Step 1: The draft model (typically 1-7B parameters) generates N candidate tokens autoregressively. This is fast because the model is small.
• Step 2: The verifier model (the full large model) processes all N draft tokens in parallel as a single batch. Thanks to the attention mechanism, verifying N tokens costs roughly the same as generating 1 token.
• Step 3: The verifier accepts tokens that match its own distribution (using rejection sampling) and discards the rest. Generation continues from the last accepted token.

The key insight: verification is parallel but generation is sequential. If the draft model proposes 8 tokens and 6 are accepted, you've effectively generated 6 tokens for the compute cost of ~1.5 large-model forward passes instead of 6.

Acceptance Rate: The Critical Metric

The speedup depends entirely on how often the large model accepts the draft model's proposals. For code generation — where much of the output is predictable syntax, variable names, and common patterns — acceptance rates typically range from 70-85%. For creative writing or novel reasoning, rates drop to 40-60%.

This is why speculative decoding is particularly effective for coding tasks. A function that starts with def process_data(self, input_data: List[Dict]): is highly predictable token-by-token once the intent is established. The draft model nails these patterns, and the verifier confirms them in bulk.

Block-Diffusion: The MiMo-DFlash Approach

Standard speculative decoding still generates draft tokens one at a time — it just uses a smaller model to do it faster. Block-diffusion speculative decoding, as implemented in Xiaomi's MiMo-DFlash, takes this further by generating entire blocks of draft tokens in parallel.

Instead of a small autoregressive draft model, block-diffusion uses a Transformer-based diffusion model that generates an entire block (32-64 tokens) simultaneously through iterative denoising. The process looks like:

• Start with random noise in the shape of N token embeddings
• Denoise iteratively (3-5 steps) to produce coherent token sequences
• Verify the entire block against the large model in one pass

The advantage: drafting 64 tokens takes roughly the same wall-clock time as drafting 8 tokens autoregressively, because all positions are filled in parallel. The tradeoff is slightly lower acceptance rates (60-75% for code) since parallel generation can't condition each token on previous draft tokens.

How This Translates to Cost Reduction

Speculative decoding doesn't reduce the hardware required — the large model still needs its full GPU allocation. What it does is multiply throughput. If the same hardware serves 4x more tokens per second, the amortized cost per token drops proportionally. For API providers, this means:

• Standard autoregressive: 1x throughput, baseline cost per token
• Speculative decoding (standard): 3-4x throughput for code, 2-3x for general text
• Block-diffusion speculative: 4-6x throughput for code, 3-4x for general text

This is why models like Tencent Hy3 can offer pricing at $0.14/$0.58 per million tokens despite having 295B total parameters. MoE architecture reduces active compute, and speculative decoding multiplies throughput on that reduced compute. The combination makes sub-$1 per million tokens viable on frontier-class models.

What This Means for Developer Budgets

As speculative decoding becomes standard infrastructure (it's already deployed by Anthropic, Google, and most Chinese labs), the pricing pressure intensifies. Models that cost $10/$50 today (like Claude Fable 5) may see output costs drop as providers optimize their serving stack. The pattern from the last 18 months is clear: serving optimization, not model training, is the primary driver of price reductions.

For developers budgeting AI coding costs, the practical implication is that per-token prices will continue falling 30-50% annually even for the same model. Lock in workflows and volume commitments rather than optimizing for today's exact pricing — the trajectory favors patience.

Limitations and When It Doesn't Help

Speculative decoding has diminishing returns for tasks requiring high novelty (every token is unpredictable), very short outputs (overhead of draft/verify exceeds savings), or when the draft model is poorly matched to the domain. It also doesn't help with input processing costs — prompt tokens are already parallelized. The technique specifically targets the output generation bottleneck.