Interleaved thinking: effort knobs vs tool-loop input tax

The last lesson treated a reasoning model’s hidden thinking as its own bill. Now put that model inside an agent that calls tools: search, run code, check a calendar. It thinks, calls a tool, reads the result, thinks again. And here’s the twist: the earlier thinking often gets sent back in with every next step. Thinking becomes input, and input piles up.

How interleaved thinking works

Interleaved thinking means the model can start a fresh thinking block after each tool result, rethinking its plan instead of committing to one at the start. Anthropic’s tool-use walkthrough shows the shape: think, call a tool, get the result, think again, maybe another tool, then the final answer. OpenAI’s reasoning models do the same between tool calls on the Responses API. Gemini keeps continuity with “thought signatures.”

All three share one rule: you pass the thinking back with the tool result.

Anthropic wants the assistant’s content echoed back unmodified, or the request errors with a 400.
OpenAI wants reasoning items, function calls and tool outputs since the last user message.
Gemini wants thought blocks and signatures resent (or kept server-side in stateful mode).

Where the cost comes from

Once those blocks sit in the history, they’re input on the next call. On Anthropic, during a tool loop the earlier thinking can be cached and then read as input tokens. Newer Claude behavior can keep prior-turn thinking in context and bill it as input, with “clear thinking” options to keep only the last N turns. OpenAI’s GPT-5.6 family defaults to reasoning.context=all_turns, which feeds earlier reasoning into the next step and raises token use on multi-turn work even if your app didn’t change. Setting current_turn restores the older behavior.

OpenAI’s usage report already separates the hidden share: their sample shows 1024 reasoning tokens inside 1186 output tokens. Turn effort up and keep all turns on a tool-heavy agent, and you buy more reasoning per step and more of that history fed into later steps.

The knobs

Anthropic: adaptive thinking steered by an effort setting (low through max). Effort shapes thinking, number of tool calls and preamble length together; lower effort tends toward fewer, terser tool calls.
OpenAI: reasoning.effort (none through max, model-dependent), plus max_output_tokens as a hard cap. They recommend about 25k tokens of headroom to start.
Gemini: thinking_budget in 2.5 (-1 means dynamic), thinking_level in 3.x (minimal, low, medium, high).

The hidden server-side trap

Servers reuse work on prompts that start the same way (prefix caching). If the server caches the thinking tokens but the next request strips the thinking out, the cached chain never matches again. A vLLM issue works the math: about 5,000 thinking tokens per turn is about 1.3 GB of dead cache on QwQ-32B and about 1.6 GB on DeepSeek-R1-Distill-70B. Fifty conversations each holding one dead chain lands on the order of 65 to 80 GB of wasted memory. A proposed fix to stop caching thinking claims about 7.6% lower peak cache use. SGLang hit the same problem and merged a fix. If you do keep thinking in later prompts, you want the opposite setting. Cache policy has to match how the next prompt is built.

An analogy: the meeting minutes

Imagine a project meeting where, before each new agenda item, someone reads aloud the full minutes of every previous discussion. Item one is quick. By item ten, you spend most of the meeting re-listening to old minutes. You can read only the last two items’ minutes (strip old thinking), have the group think less hard about easy items (lower effort), or accept the long meeting because continuity matters (keep everything and pay). Teams argue about this exact trade.

Try this

Have a long, multi-step conversation with a chatbot that shows token counts or usage (many developer playgrounds do). Ask it to plan a trip, then refine it step by step for ten messages. Watch the input token count grow each turn even though your messages stay short. That climbing number is the “input tax.” Now start a new chat and paste only a short summary of the plan. Notice how much smaller the input is.

Use it at dinner

“AI agents that think between every step often re-read all their old thinking each time. It’s like re-reading the full meeting minutes before every agenda item.”