Speculative decoding is an inference technique where a small draft model guesses several upcoming tokens first, and the large target model verifies them all at once to make LLM responses faster. When a large language model generates tokens one after another, GPU compute units spend much of the time sitting idle while waiting to read model weights from memory; this post explains why doing more total computation still produces faster answers and when the speedup shrinks.
Summary
Speculative decoding is a method where a small draft model guesses several upcoming tokens first, and the large target model verifies that draft in a single forward pass, accepting only the tokens it would have produced itself. Although total computation actually increases, generating one token at a time leaves GPU compute units idle while waiting to read model weights from memory; using that idle compute to verify drafted tokens makes overall generation faster. As a result, the speedup is greatest when there are few concurrent requests, and shrinks when batches are large. By design, output quality follows the large model’s distribution.
What is speculative decoding?
One-line answer: Speculative decoding is an inference acceleration technique where a small draft model proposes several next tokens ahead of time, and the large target model verifies them in a single forward pass to accept the matching ones.
Standard decoding generates one token at a time sequentially using the large model. In contrast, speculative decoding works through a cycle of drafting, verification, and acceptance or rejection:
- Draft: A small, fast draft model proposes several next tokens in advance.
- Verify: The large target model checks all of those drafted tokens simultaneously in a single forward pass.
- Accept or reject: The large model keeps the tokens it would have produced itself; at the first rejection, it substitutes its own token and drafting restarts from there.

Both the Hacker News discussion “Why Is Speculative Decoding Fast?” and the original article Why Is Speculative Decoding Fast? by Joe Barrow walk through how this draft-and-verify loop translates into faster wall-clock generation.
Why does it get faster when total computation increases?
One-line answer: When generating tokens one by one, GPU compute units sit idle waiting to read model weights from memory (memory bound), and speculative decoding uses that idle compute to verify several drafted tokens during the same memory read.
Running a separate draft model and verifying tokens that might end up rejected actually increases total computation. The reason overall latency still drops is that generating a single token at a time is memory bound rather than compute bound.
Each time the large model produces a token, the GPU must read the model weights from memory, leaving the compute units mostly idle while waiting for data movement. Think of a delivery truck that takes roughly the same trip time whether it carries a single box or several boxes at once: reading the model weights from memory is the trip, and verifying several drafted tokens in the same pass uses the empty space in the truck for roughly the same memory reads.
Speculative decoding is not a replacement for KV caching and works alongside it; for an overview of standard inference and caching, see Day3: LLM Inference Structure and KV Cache (Korean).

When is speculative decoding less effective?
One-line answer: The benefit shrinks when large batches already keep the GPU compute units full (compute bound), or when the draft model is frequently wrong and yields a low acceptance rate.
Speculative decoding helps less in two main situations:
- Large batches (compute bound): With large batches of concurrent requests, the GPU is already compute bound—its compute units are full—so there is little idle compute left and the gain shrinks.
- Frequently wrong drafts (low acceptance rate): When the draft model often guesses wrong, the rejected tokens turn into wasted work.
Because of this trade-off, serving engines have settings to turn speculative decoding off depending on batch size.
Does the output quality change?
One-line answer: In the original speculative decoding method’s design, the verification and acceptance rule is constructed so that the output follows the large model’s distribution, preserving output quality.
Having a small model draft tokens does not degrade the final response to the small model’s level. Because the target model checks every drafted token and replaces the first rejected token using its own distribution, the output follows the large model’s distribution in the original method’s design (without asserting exact guarantees for every individual implementation).
What are the frequently asked questions?
One-line answer: Here is how speculative decoding relates to sampling settings, model quantization, and local inference frameworks.
How does it relate to temperature and top-p?
The sampling setting stays the same, as the verification rule accounts for the chosen sampling distribution. For background on sampling parameters, see What are Temperature and Top-p? Why LLM answers vary every time.
How is it different from quantization?
Quantization makes the model weights smaller, whereas speculative decoding changes how the model runs at inference time, and in general the two can be combined. For an introduction to quantization, see What is Quantization?.
Can it be used in local environments?
Yes, some inference frameworks support speculative decoding for local execution. For an overview of inference engines, see What is an LLM Inference Framework?.
What related posts are worth reading?
One-line answer: Here is a list of related posts covering LLM inference architecture, sampling, model compression, and hardware differences.
- Day3: LLM Inference Structure and KV Cache (Korean)
- What are Temperature and Top-p? Why LLM answers vary every time
- What is Quantization?
- What is an LLM Inference Framework?
- What is the difference between NPU and GPU? Easy comparison of architecture, use cases, and power
What are the sources?
One-line answer: Here are the original article and community discussion sources referenced in this post.
- Joe Barrow — Why Is Speculative Decoding Fast? (checked 2026-10-28)
- Hacker News — Why Is Speculative Decoding Fast? (checked 2026-10-28)