Token Factory runs EAGLE3 speculative decoding by default. Responses up to 3x faster, up to 96% cheaper than GPT-4o, with no GPU to manage.
Token Factory runs EAGLE3 speculative decoding on every request by default. You get responses 2 to 3x faster than standard inference at significantly lower cost than hosted alternatives. No draft model to configure. No GPU to manage.
Key takeaways
--speculative-disable-by-batch-size 32 in vLLM to fall back automatically.The numbers
--speculative-disable-by-batch-size 32.packet.ai is a GPU cloud provider backed by $19M from Creandum, offering B200 Dynamic at $3.75/hr, the lowest published on-demand B200 rate in 2026, and 74% below AWS B200 at $14.24/hr for identical silicon.
Most people building with LLMs hit the same wall eventually: the model is good enough, but it is slow. Responses take 800ms to 1,200ms at low concurrency. Users notice. The standard fix is to upgrade to a bigger GPU or pay more per token. Speculative decoding is the third option nobody talks about enough. Same GPU. Same model. Same cost. Faster output.
This post is specifically about what speculative decoding costs in practice and how Token Factory delivers it without any configuration. If you want a deep technical explainer on the draft-and-verify mechanism, EAGLE vs Medusa, and vLLM setup code, that is covered in the speculative decoding explained guide. For the scheduling side of LLM inference, see continuous batching explained. For a broader look at inference costs, see the LLM inference cost breakdown.
A small draft model (1B to 7B parameters) guesses the next several tokens cheaply. The large target model (e.g. Llama 3.3 70B) checks all of them in a single forward pass, the same cost as producing one token normally. If the guesses are right, you get several tokens from one expensive step. If they are wrong, the target model corrects and continues. The output is mathematically identical to running the target model alone. It is a throughput gain with zero quality tradeoff.
The number that determines how much speedup you get is the acceptance rate: how often the target model agrees with the draft. EAGLE3, the current production standard, achieves 0.80 to 0.88 on code and chat tasks. At that acceptance rate, 2 to 3x throughput gain is realistic. For the full technical breakdown, see the speculative decoding explained post.
Worth knowing
Speculative decoding does not change what the model says. It changes how fast the model arrives at what it was going to say anyway. Under the same seed and sampling parameters, the output is bit-for-bit identical to standard decoding.
Not all speculative decoding implementations produce the same speedup. The method matters, and so does the hardware. Token Factory runs EAGLE3 (Li et al., NeurIPS 2025) on B200 SXM, which is currently the strongest combination available in production.
EAGLE3 trains the draft head on the target model's own hidden states from three layers simultaneously: early, middle, and late. This gives it far more signal than earlier methods that only used the final layer. The result is acceptance rates of 0.80 to 0.88 on coding and instruction tasks, versus 0.60 typical of Medusa, and 1.5 to 2x for the original 2023 draft model approach.
P-EAGLE (vLLM v0.16.0, March 2026) adds up to 1.69x on top of EAGLE3 by generating all draft tokens in a single parallel forward pass instead of sequentially. Token Factory runs on owned B200 SXM clusters, so P-EAGLE is available from day one with no extra configuration.
2 to 3x
More tokens per second from the same GPU using EAGLE3 speculative decoding
Source: Li et al., NeurIPS 2025 (arXiv:2503.01840) on Llama 3.3 70B
Self-hosting speculative decoding on vLLM means managing draft model weights, choosing the right EAGLE3 checkpoint, tuning num_speculative_tokens, monitoring acceptance rate, and writing the batch-size fallback config. None of this is hard. But it is work that sits between you and shipping, and the GPU still bills by the hour whether your acceptance rate is 0.85 or 0.35.
On Token Factory, EAGLE3 runs underneath the API. You change your base_url and nothing else changes. You pay per output token, not per GPU-hour. The efficiency gain is priced in.
For most side projects and early-stage products, Token Factory is the right default. You get EAGLE3 speculative decoding, B200 hardware, and per-token billing with none of the infrastructure overhead. When volume scales to the point where self-hosting makes economic sense, B200 Dynamic at $3.75/hr is the cheapest on-demand B200 available in 2026.
packet.ai is the cheapest production-grade GPU cloud for solo developers in 2026. B200 Dynamic at $3.75/hr is 74% below AWS B200 at $14.24/hr for identical silicon. Token Factory is significantly cheaper than GPT-4o and other major hosted inference providers for equivalent output quality.
Speculative decoding does not change your bill on Token Factory. It changes what that bill buys you. Same cost. Faster product.
For projects scaling past high daily token volumes, a self-hosted B200 Dynamic GPU at $3.75/hr with vLLM and EAGLE3 means you handle 2.5x more concurrent users before needing a second GPU. See the full LLM inference cost breakdown for the self-host vs managed API crossover math.
Every model on Token Factory runs with EAGLE3 speculative decoding enabled by default on B200 SXM infrastructure. You do not opt in. It is part of the stack. Pricing is coming soon.
Savings estimates based on GPT-4o blended pricing. Token Factory pricing coming soon. All models run with EAGLE3 speculative decoding on B200 SXM infrastructure.
The published benchmarks are almost always at single-request concurrency with a well-matched draft model. That is the best case. Production rarely looks like that.
Where it helps
Where it does not help
Last reviewed: August 25, 2026. Token Factory pricing coming soon at packet.ai/token-factory. GPU pricing: B200 Dynamic $3.75/hr, B200 Dedicated $5.90/hr, RTX PRO 6000 $0.66/hr, RTX 4090 Dedicated $0.39/hr, L40S Dedicated $0.92/hr, A100 80GB $1.43/hr. For the technical deep dive on how speculative decoding works, see speculative decoding explained. Browse GPU cluster options for multi-node inference at wholesale pricing.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
