Treat draft length as a live control; a fixed max draft burns compute once the batch is full.

The speculative decoding lesson showed that a fast guesser plus a careful checker can speed up AI responses, but only when the servers aren’t slammed. This lesson is the natural follow-up: if the right amount of guessing depends on how busy you are, why is it usually a fixed setting?

The problem: one knob, set once

Many production engines still ship a static max draft length: how many tokens the guesser proposes each step. TensorRT-LLM’s own docs say it plainly: with speculation on, every request gets the same draft length, and there is currently no way to dynamically turn speculation off, so speedups show up mainly at low batch sizes.

SpecDecode-Bench (2026) measured five speculative variants on production vLLM (H100s, continuous batching, CUDA graphs):

At batch size 1, EAGLE reaches up to about 1.96x token throughput on Llama-3-70B.
By batch 128, gains shrink (about 1.21x on one 8B math run).
Verification takes 42 to 95% of runtime.
Wide guess trees look clever at batch 1, then a k=21 tree falls under 1x (slower than no speculation) by batch 64, because rejected branches still cost the big model work.

The familiar failure: a demo number measured on an empty server gets wired in as a permanent setting, then midday traffic turns speculation into wasted checking and jittery response speed.

The fix: make it a live decision

AdaSpec / SpecServe (SoCC 2025) decides draft length on the fly. It looks at the current load (batch size, context length) plus how confident the guesser is, and checks whether a given draft length will still meet the per-token speed target before the step runs. A filter also drops low-odds guesses so the big model does less useless checking. On real traffic traces on top of vLLM, they report up to about 66% speedup versus earlier speculative serving setups while keeping speed targets met.

WISP pushes the same idea into setups where a phone or edge device guesses and a cloud server checks. It names two bottlenecks: Wasted Drafting Time (the device keeps guessing past the first mistake) and Verification Interference (short checks get stuck waiting behind long ones in the same batch). Speed targets come in classes like 2, 4, 6 and 8 tokens per second. A predictor stops guessing early when a rejection is likely. Versus centralized serving, they claim up to about 2.1x capacity and 1.94x useful throughput; versus an earlier system called SLED, up to about 4.1x capacity.

An analogy: cruise control versus a fixed throttle

Imagine taping your car’s gas pedal at one position that worked great on an empty highway at 6am. At rush hour you’d be slamming into traffic and braking constantly. Adaptive cruise control reads the car ahead and adjusts. Same engine, much better trip.

A fixed draft length is the taped pedal. Adaptive draft length is cruise control.

Try this

Revisit the guessing game from the speculative decoding lesson, with one change. Each round, before you guess, decide how many words to guess (1, 3 or 5), based on how predictable the next bit looks. Score +1 per correct word and -1 for every wrong guess. Play one round with a fixed 5 every time, and one round adjusting. On anything but a very familiar text, the adjusting round almost always wins. That’s AdaSpec in a living room.

Use it at dinner

“The ‘guess ahead’ trick in AI serving is usually set once, like taping the gas pedal. The newer systems treat it like cruise control and back off when the servers get busy.”