Speculative decoding pays only when you measure it at real concurrency and on the actual task mix.

Models write one token at a time, and each step means reading through the whole model. Speculative decoding is a trick to take fewer steps without changing the answer. It’s one of the most popular speedups in AI serving. It also has a habit of looking amazing in demos and quietly losing money in production.

How the trick works

A cheap proposer guesses the next few tokens. The real model then checks all those guesses in a single pass, instead of one pass per token.

Guesses it accepts are free speedup.
Guesses it rejects cost the checking work for nothing.

Quality stays the same, because the real model always has the final say.

Why busy servers change the math

At low traffic, the GPU has idle capacity, so a wasted guess costs little. Pack 200 requests onto the same GPU and every rejected guess steals compute a real request needed.

The authors of AgentSpec report that earlier speculative methods ran slower than plain decoding above batch size 32 on one Qwen-3-8B workload.

A newer twist for AI agents

AI agents have structure: the model is writing code, or making a tool call, or reading a result back. AgentSpec (a new preprint) feeds those boundaries to the serving system so it can guess better.

It uses no separate draft model at all. It pulls candidate continuations from earlier text in the same request (agents repeat themselves a lot), and gives more guessing budget to requests with reusable text. That budget shrinks as the batch grows.

Numbers, all reported by the authors on their own vLLM v0.12.0 setup:

GPT-OSS-20B code generation on one A100 at max batch 256 went from 289 to 584 tokens per second, about 2.02x, with worst-case (P99) end-to-end latency 1.39x better.
On a Qwen-3-8B Reflexion workload, rejected guesses fell to 26.4%, versus 53.3% for EAGLE-3 and 65.9% for SuffixDecoding.

It’s one preprint with no independent replication yet. Hold it loosely.

An analogy: the eager intern

You’re writing a report. An intern reads over your shoulder and types ahead what they think your next five words will be. If they’re right, you just nod and keep going: five words for the price of one glance. If they’re wrong, you had to read their guess and delete it.

When you’re alone in a quiet office, an eager intern is great. When you’re in a packed meeting with forty people waiting on you, every bad guess is time stolen from someone. A good intern learns when to stay quiet.

What to measure

Token acceptance rate only tells you how good the guesser is. What actually matters is useful work delivered at your real traffic levels, the slow-tail latency (P95 and P99), and cost per successful task. A method can raise acceptance and still lose money.

The live debate: learned draft models like EAGLE-3 win at low traffic (their paper reports 1.38x throughput at batch 64 in SGLang). History-based methods like SuffixDecoding skip the draft model and feed on repetition. AgentSpec claims both fall apart in big agent batches. The winning policy probably switches methods, or turns speculation off, as traffic changes.

Try this

Play the intern game with a friend. One of you reads a paragraph aloud slowly. The other tries to say the next three words before the reader gets there. Score: +3 for each correct guess, -1 for each wrong one (that’s the wasted check). Try it once with a familiar text (a nursery rhyme) and once with something unfamiliar (a technical manual). You’ll see in about a minute why speculation is great on repetitive text and wasteful on surprising text.

Use it at dinner

“Speculative decoding is like letting a fast guesser type ahead while the real model just checks. It’s great when the servers are quiet and the text is predictable, and it can burn compute when they’re slammed.”