Agent loop compaction economics (cache thrash vs prune)

Long AI agent sessions grow on every step. The obvious move is to summarize (“compact”) the history so you send fewer tokens. Here’s the surprise: that often costs more, because it throws away the discount you were getting for sending the same thing again.

The discount: prompt caching

If you resend the exact same beginning of a prompt (system instructions, tool definitions, history), providers can reuse their work and charge a cache-read rate. On Anthropic and Bedrock, a cache read is typically about 10% of normal input price. A cache write is about 1.25x input, and a 1-hour cache write is often about 2x. Gemini’s cache reads are about 90% off. Cost and wait time stay friendly as long as the beginning stays byte-for-byte identical.

Three ways to shrink

Keep-all: resend everything and ride the cache.
Context editing: Anthropic’s tools for clearing old tool results or old thinking blocks before the prompt reaches the model. Clearing breaks the cache from the edit point on, so there’s a clear_at_least setting to avoid paying a cache miss for a tiny trim.
Server compaction: the heavier move. Anthropic’s version (default threshold about 150k input tokens, floor 50k) replaces older history with a compaction block. OpenAI’s Responses API has a compact_threshold setting and a standalone compact endpoint that returns an encrypted compaction item.

Compaction rewrites the beginning of the prompt, so the next turn often pays full price again.

What the measurements say

Anthropic’s internal agentic-search eval: context editing alone at +29% versus baseline, and editing plus their memory tool at +39%. In a 100-turn web-search eval, editing cut token use about 84% and finished runs that otherwise ran out of room.
Louis Bouchard and Towards AI’s tutor evals (2026): keep-all beat summarization for cost, speed and remembering planted facts. Their compacting preset sent about 41% fewer tokens and still paid about 2x, because keep-all got about 87% of input at cache rates while every summary rewrote the prefix.
Capping tool outputs at a fixed size, no model call needed: about 38% lower cost per turn, cache hit rate staying around 96%, memory tests unchanged.
A cache-aware summarize (keep history unchanged and add the summarize instruction at the end) lifted the summarizer’s cache hit rate from near zero to about 94% and cut summarize cost about 87%, though it still landed about 14% above capped keep-all.
Their rule of thumb: keep-all wins while cached input costs under roughly $0.55 per million tokens.
One analysis of trimming on Claude: a one-time cache miss from a rewrite often breaks even in about 3 to 10 tool-heavy turns; quiet chat sessions can need 30 or more.

Claude Code’s team treats cache hit rate like uptime and gets alerted when it breaks. Their compaction trick keeps the system prompt, tools and history identical and adds the “please summarize” instruction as the last message, so the summarize call itself hits the cache.

When pruning does win

Shrink when you’re about to run out of context window, when stale tool junk is hurting quality, or when your cached-input price is high enough that resending a huge prefix loses. Try prefix-friendly shrinks first: cap tool outputs, clear old results in bigger chunks, keep recent thinking. Save full summaries for natural task boundaries.

An analogy: the photocopier with a discount

Imagine a copy shop that charges 10 cents a page for new documents but 1 cent a page for anything it has already scanned, as long as every page is in the same order. You have a 100-page binder you need copied each day. Keeping it as-is costs $1. Rewriting it into a tidy 40-page summary costs $4 the first day because it’s all new, and the summary might lose something. Summarizing only pays when the binder is getting too big for the shop, or when you’d otherwise keep it for weeks.

Try this

Grab the pricing page for any AI API that lists cached-input prices. Do the photocopier math: a 50,000-token history, plus 1,000 new tokens per turn, for 10 turns. Price it once as keep-all (the history at the cache-read rate) and once as “summarize to 10,000 tokens on turn 1” (that summary at the full or cache-write rate the first time). Which wins? Now try 50 turns.

Use it at dinner

“Summarizing a long AI chat to save money can backfire. Providers give you a big discount for resending the exact same text, and a summary is brand-new text.”