Reasoning decode economics: thinking tokens, relaxed acceptance, SLAs

When a “reasoning” model pauses before answering, it isn’t idle. It’s writing, often a lot, in a hidden scratchpad. Those thinking tokens are generated exactly like the answer tokens you see. They count toward how long you wait, what you’re billed, and how many other people the GPU can serve at the same time.

How different labs handle the hidden part

OpenAI bills reasoning tokens as output and keeps raw reasoning out of the API text. The usage report shows the count in output_tokens_details.reasoning_tokens.
Anthropic bills thinking as output even when display is turned off.
Open R1-style models write a long <think> … </think> block before the answer.

In every case, the GPU generates those tokens before (or while) you see anything useful.

Two phases of writing

A reasoning request has a thinking phase that can run to hundreds or thousands of tokens (DeepSeek-R1-Zero’s thinking grew into that range as its training rewarded longer reasoning), then a shorter visible answer. OpenAI recommends reserving at least about 25,000 tokens of headroom when you first size max_output_tokens for reasoning plus visible output.

A speed trick only for the thinking

Speculative decoding (a fast guesser plus a checker) still works here, and the checking rule can differ by phase. TensorRT-LLM’s relaxed acceptance for DeepSeek-R1 applies only inside the thinking tags: a guess can pass if it’s among the top few likely tokens, not only the single top one. After </think>, checking goes back to strict.

On DeepSeek-R1-FP4 (batch 1, 8 B200 GPUs, 1K-token prompts and 2K-token outputs), they report 2.16x speed with MTP guessing 3 tokens versus none, and 2.33x with relaxed acceptance during thinking. Their tests show a slight accuracy drop. The speedup is the point.

The hosting cost difference can be large. SemiAnalysis’s comparison of DeepSeek R1 on H100 FP8, at about 26 tokens per second per user, puts speculation on near 383 tokens per second per chip and about $0.85 per million tokens, versus about 96 and $3.39 with it off.

Three different clocks

“How fast is it?” splits into three questions for reasoning models:

Time to first token on the wire, which can be early if you stream the thinking.
Time to the first visible answer, which comes later if thinking is hidden.
Time to final, dominated by how long the thinking runs.

Anthropic’s docs spell out the trade: turning off thinking display gets the first visible text sooner, and you still pay for all the thinking tokens. Cost and latency are different knobs.

The debate

One camp says thinking quality barely moves the final answer for many tasks, so loosen checking during thinking, cap the thinking budget, and treat it as a tax to shrink. The other says the reasoning is the product, so lossy shortcuts will show up in quality. Both are partly right. One concrete lesson from vLLM: there was a stretch where the per-request thinking_token_budget was silently ignored when MTP speculation was on, until a later fix. Test your limits with your speed tricks switched on.

An analogy: the math test

Picture a student taking a test who fills three pages of scratch paper, then writes one line as the answer. The teacher only grades the final line, but the student still spent the whole period writing. If the school charged for paper, the scratch pages would be most of the bill. Reasoning models are that student. The scratch paper is real work, and you’re paying for it.

Try this

If you have access to a reasoning model with an effort or thinking setting, ask the same moderately hard question (a logic puzzle works) at the lowest and highest settings. Time both, and if your app shows token usage, compare the counts. Then compare the answers. Was the extra thinking worth the wait for this question? For many everyday questions, it isn’t. For hard ones, it often is.

Use it at dinner

“Reasoning models write a hidden essay before every answer. You usually can’t read it, but you pay for it and you wait for it.”