A small draft model guesses ahead, the big model checks its work in one pass. Here is why that is lossless, what makes it fast, and how to set it up in vLLM.
Speculative decoding speeds up text generation by letting a smaller model guess a few tokens ahead while the larger model verifies them in one pass. If the guesses are accepted, generation moves forward several tokens at once instead of one at a time. The important part: the final output doesn't change. vLLM supports several ways to generate those guesses, from a separate draft model to n-gram matching to EAGLE-style feature-level drafting.
Key takeaways
speculative_config dictionary, with a method key selecting between a standalone draft model, n-gram matching, or EAGLE-style drafting"Speculative decoding" describes a family of techniques, not one specific method, and the differences between them matter more than the shared name suggests. This post covers speculative decoding LLM inference from first principles: the core draft-and-verify mechanism, why it's mathematically lossless rather than a quality tradeoff, how the major variants (EAGLE, Medusa, n-gram, lookahead) actually differ from each other, and how to configure it in vLLM. This is part of the packet.ai LLM serving frameworks cluster; for the memory-management side of vLLM's internals, see the vLLM Docker deployment guide, and for the scheduling side, see Continuous Batching Explained.
Autoregressive generation is memory-bound, not compute-bound: producing one token means one full forward pass through the entire model, and each pass mostly waits on memory bandwidth rather than saturating GPU compute. Producing 100 tokens means doing this 100 times, sequentially, each one dependent on the last. The GPU spends most of that time under-utilized.
Speculative decoding restructures this. Instead of generating one token per forward pass, it runs a loop: a small draft model proposes several tokens ahead (commonly 3-8), then the large target model checks all of those proposed tokens in a single forward pass, the same cost as generating just one token normally. The target model accepts the longest prefix of draft tokens it agrees with, then generates one further token itself to keep the sequence moving forward. Whatever the draft model got wrong is simply discarded, and the process repeats from the point of disagreement.
The technique traces to Stern et al. (2018), who first proposed the draft-then-verify pattern, later formalized into speculative sampling by Leviathan et al. (2023) and Chen et al. (2023) at DeepMind. The core guarantee those papers established, and the reason this isn't a quality-for-speed tradeoff: the target model verifies every draft token against its own probability distribution, and the sampling procedure is constructed so the output distribution is provably identical to sampling from the target model alone.
In this example, the draft model proposes five tokens continuing "Hel-lo-my-nam-xyz." The target model checks all five in that single pass: it agrees with the first three, disagrees at the fourth, and never bothers checking the fifth since everything after a rejection gets discarded anyway. The target model then generates the correct token in that fourth position itself, and the draft model starts proposing again from there. Three tokens' worth of progress came from one expensive forward pass instead of three.
The entire speedup depends on one thing: how often the target model agrees with what the draft model proposed. If the draft and target models make similar predictions most of the time, the target model accepts long runs of draft tokens per verification pass, and each expensive forward pass produces many tokens' worth of progress. If the draft model's guesses are frequently wrong, the target model accepts only one or two tokens before rejecting the rest, and the draft model's extra compute was wasted.
This is why "just pick any smaller model as the draft" is the wrong instinct. A draft model that's fast but poorly aligned with the target model's behavior can end up slower than not using speculative decoding at all, since you're now paying for both the draft model's forward passes and a target model verification pass that only accepts a token or two each time. The draft model needs to actually predict what the target model would say, not just be small.
⚡ Note
Acceptance rate also varies by task, not just by method. The same draft/target pair can show a high acceptance rate on code completion, where continuations are more predictable, and a much lower one on open-ended creative writing. Benchmark on traffic that resembles what you'll actually run, not a generic prompt set.
The original formulation of speculative decoding used a genuinely separate, smaller language model as the draft model, an actual second model with its own weights, run independently. That still works, but most of the practical innovation since 2023 has been in finding faster or more accurate ways to generate draft tokens without the overhead of a full second model.
The practical pattern: EAGLE-3 is the strongest general-purpose choice when you can use a compatible draft checkpoint, Medusa is simpler to reason about but leaves acceptance rate on the table, and n-gram or lookahead approaches are worth trying first if your workload has heavy repetition and you'd rather not manage a second model at all.
vLLM speculative decoding is configured through a single speculative_config dictionary rather than a scattered set of flags. The method key selects the drafting strategy; everything else follows from that choice.
# Separate draft model
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.1-70B-Instruct",
tensor_parallel_size=4,
speculative_config={
"model": "meta-llama/Llama-3.1-8B-Instruct",
"num_speculative_tokens": 5,
},
)
# N-gram speculation: no draft model needed
llm = LLM(
model="meta-llama/Llama-3.1-70B-Instruct",
speculative_config={
"method": "ngram",
"num_speculative_tokens": 5,
"prompt_lookup_max": 4,
},
)
# EAGLE-style, feature-level drafting
llm = LLM(
model="meta-llama/Llama-3.1-8B-Instruct",
tensor_parallel_size=4,
speculative_config={
"model": "yuhuili/EAGLE-LLaMA3-Instruct-8B",
"method": "eagle",
"num_speculative_tokens": 2,
"draft_tensor_parallel_size": 1,
},
)One constraint worth knowing before you configure this: EAGLE-based draft models must run without tensor parallelism of their own, draft_tensor_parallel_size stays at 1, even when the target model is sharded across multiple GPUs with a higher tensor-parallel-size. The draft head is small enough that splitting it across GPUs adds coordination overhead without a meaningful memory benefit.
Two things to check before committing to a configuration in production:
Speculative decoding's payoff depends heavily on GPU memory headroom and batch size, both of which are easier to experiment with when the hardware itself isn't a fixed commitment. packet.ai's Dynamic tier bills hourly rather than requiring a fixed allocation, which makes it a reasonable place to actually test a draft/target pairing, or compare EAGLE against Medusa against no speculation at all, across a range of concurrency levels before deciding what belongs in production. H100 and B200 instances both have the memory headroom to run a 70B target model alongside a small draft model without immediately hitting the concurrency tradeoff described above. Once you've confirmed which configuration actually helps your traffic, moving to a Dedicated instance locks in that setup without scheduler variance.
Last reviewed: August 6, 2026. For the memory-management side of vLLM's internals, see the vLLM Docker deployment guide. For the scheduling side, see Continuous Batching Explained.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
