Pick the speculative proposer from traffic shape and packaging cost; score tokens/sec under your concurrency, not paper acceptance.

Speculative decoding is the same loop every time: a cheap proposer guesses the next few tokens, the real model checks them all in one pass, and you keep the longest run of correct guesses. (See the speculative decoding lesson for the basics.) The interesting question is which guesser to use. Three show up in every serving stack.

Option 1: n-gram, or prompt lookup

It copies continuations it has already seen in the prompt or earlier output. No extra model, no extra memory. TensorRT-LLM and vLLM both treat it as the free option for text that overlaps the input: summarization, answering questions about a document, code edits, JSON that echoes a schema, chats that reuse earlier turns.

When the answer barely overlaps the context, guesses mostly fail, and you’ve paid for wasted checking.

Option 2: Medusa

Medusa attaches a few small extra “heads” to the same model, so one pass proposes several future positions at once, then checks a tree of candidates together.

Medusa-1 freezes the main model and trains only the heads. It stays exactly faithful to the original model’s output.
Medusa-2 trains the heads together with the model, aiming for more accepted guesses.

The ICML paper lands around 2.2x speedup for Medusa-1 and roughly 2.3 to 2.8x for Medusa-2 on Vicuna-class chat. The practical win is packaging: one file, no second model to manage. The catch: a looser acceptance rule can trade exact faithfulness for longer accepted runs. If you need output identical to the original model, stick to strict checking.

Option 3: EAGLE-3

EAGLE-3 is the current high bar for trained guessers on general chat and reasoning. It predicts tokens directly, combines information from low, middle and high layers of the main model, and is trained in a way that keeps multi-step guesses on track. NeurIPS 2025 numbers: up to about 6.5x wall-clock speedup versus normal decoding on HumanEval with Vicuna-13B, roughly 1.4x over EAGLE-2, and about 1.38x throughput at batch 64 in SGLang.

A practical gotcha: in vLLM you must set the method to eagle3 with a matching draft checkpoint; loading an EAGLE-3 head as eagle fails on weight shapes. And training the guesser needs its own training data, which is a real ongoing cost.

How to choose

Outputs mostly quote or rearrange the prompt? Turn on n-gram first.
Need general open-ended speed and can host or train a guesser? Prefer EAGLE-3 over just using a random smaller model.
Want the simplest packaging in one file? Medusa still earns its seat.

Whatever you pick, never sell the paper’s batch-1 speedup as fleet capacity. At high traffic the GPU is already busy, and checking work can erase the win (see the adaptive draft length lesson).

An analogy: three ways to finish someone’s sentence

n-gram is the friend who just repeats what you said earlier. Perfect when you’re reading a recipe back, useless when you’re telling a new story.
Medusa is your own reflex: the same brain, with a quick habit of anticipating your next few words.
EAGLE-3 is a well-trained understudy who studied you closely. Very good, but you had to train them, and you have to retrain them if you change.

Try this

Ask a chatbot to “rewrite this paragraph in simpler words” and paste a paragraph. Then ask it to “write an original paragraph about the ocean.” The first answer reuses a lot of your words; the second shares almost none. Highlight the shared words in each. The highlighted share is roughly how much an n-gram guesser could get for free.

Use it at dinner

“There are a few ways to let a fast guesser type ahead for an AI model. The best one depends on whether the answer mostly repeats your input or says something new.”