Serving agent sessions: keep the KV cache alive across tool and human gaps, route turns back to the warm replica, and price cache TTL by your pause distribution

The model routing lesson had a striking number from GitHub Copilot: cache hit rates of about 90% inside a turn, but only 55% across a turn boundary. Where do those missing 35 points go? Into the pauses. AI agents constantly stop to run a tool or ask a person something, and serving systems were built for chatbots that don’t.

How the problem happens

Serving engines like vLLM and SGLang treat a request as finished when the model stops writing. If other work is waiting, the finished request’s saved notes (its KV cache) are first in line to be freed.

For a chatbot, that’s fine: the human takes a while to reply. An agent pauses for a tool call, often about a second, then sends the same 70K-token beginning plus a few hundred new tokens. If the notes are gone, the engine recomputes them or reloads them from slower memory, and the request goes back into the queue behind everyone else.

Measuring the gaps

Continuum (UC Berkeley and Tensormesh, arXiv 2511.02230, revised May 2026) measured this. Across 100 mini-swe-agent runs on SWE-Bench with GPT-5, programs averaged 10.9 turns and about 70K tokens, and tool calls averaged 925 ms with a 3,550 ms standard deviation. The tail is extreme: the slowest 10% of cd calls accounted for 94% of total cd time, and the slowest 10% of web fetches for 52.5%.

Continuum’s fix: when the model makes a tool call, pin that session’s notes in GPU memory for a set time (a TTL, time to live). The TTL comes from a cost model weighing what a miss would cost (recompute or reload, plus the queue wait) against the memory held while waiting, using the real timing pattern for that tool. If the tool runs past the TTL, the notes are released, so a stuck install can’t hold memory hostage. Sessions that started earlier keep their place in line across turns.

The key finding is about queueing. Offloading notes to CPU memory alone saves the recompute but still leaves the queue wait, and that wait stacks up turn after turn. Across three hardware and model setups, Continuum cut job completion time 1.12x to 3.66x and raised throughput 1.10x to 3.22x. On an H100 testbed running 500 SWE-Bench Verified tasks, delay fell up to 8.18x. Scheduler overhead was single-digit milliseconds.

Others predict the gap instead: Tokencake offloads a waiting agent’s notes during a tool call and brings them back just before the call should return, cutting end-to-end latency over 47% versus vLLM. KVFlow (NeurIPS 2025) uses the workflow graph to evict whichever agent will run again latest, up to 2.19x faster than SGLang’s hierarchical cache.

Routing back to the warm machine

Keeping notes warm only helps if the next turn lands on the machine holding them. llm-d’s prefix-cache routing has every server report what it holds and sends each request to the warm one unless it’s overloaded. On 8 replicas of Qwen3-32B on 16 H100s, its July 2026 benchmark showed 14,892 output tokens per second versus 6,986 behind a plain Kubernetes Service, and p90 time to first token of 0.26 seconds versus 135.5. NVIDIA Dynamo 1.0 lets the agent send hints (priority, expected length, how long to pin), and reports up to 4x lower time to first token on Llama 3.1 with NeMo Agent Toolkit.

The pricing side: cache TTL

On Anthropic’s API, the default cache lives 5 minutes. Writes cost 1.25x base input, a 1-hour cache costs 2x to write, and reads cost 0.1x. Every read refreshes the clock for free. OpenAI’s GPT-5.6 and later default to a 30-minute minimum at 1.25x, with a 12-hour option at 2.5x.

Worked example at Claude Opus 5 list prices ($5 per million input, $6.25 for a 5-minute write, $10 for a 1-hour write, $0.50 to read). A coding agent holds 80K tokens of context and asks the user a question. The user answers 12 minutes later.

With the 5-minute cache, that turn rewrites 80K tokens: $0.50.
With the 1-hour cache, it reads them: $0.04.

The 1-hour write cost $0.30 more up front, so one long pause per session pays for it. Short tool gaps never need it, since each call refreshes the 5-minute clock.

An analogy: the restaurant table

A diner steps outside to take a phone call. Clear the table immediately and they come back to find strangers sitting there; they wait for a new table and reorder. Hold it forever and you lose covers on a busy night. A smart host holds it based on how long phone calls usually take, and seats them back at the same table, not a random one.

Try this

Think about your own AI agent use, or any chatbot with long tasks. List the last ten pauses: was the agent waiting on a tool (seconds), on you (minutes), or on something long like a build (many minutes)? Mark which ones were over 5 minutes. Using the worked example, each long pause on 80K tokens is about $0.46 extra on a 5-minute cache. Your pause list is your cache bill.

Use it at dinner

“AI agents pause all the time to run tools or wait for you. Whether the model’s short-term memory survives those pauses can matter more to the bill than which model you picked.”