Model routing and cascades for agents (price the switch; route at phase boundaries; judge on cost per success)

New models launch every few weeks, at wildly different prices. The obvious idea: send easy jobs to a cheap model and hard ones to an expensive one. That’s model routing. It works, but inside a multi-step AI agent, switching models mid-task has a cost almost nobody puts on the slide.

Three designs

Query routers read the prompt and pick a model before anything runs. RouteLLM is the reference: its router, trained on Chatbot Arena preferences, reached 95% of GPT-4’s MT-Bench score while sending only 14% of calls to GPT-4 (the cheap side was Mixtral 8x7B).

Cascades run the cheap model first, check the answer, and escalate if the check fails. The check is the hard part. UCCI turns the small model’s confidence into a calibrated error probability. On a 75,000-query production extraction workload served by 4B and 12B models, it cut cost 31% at the same accuracy (F1 of 0.91).

Agent-aware routers decide inside the run, using what the agent has already seen. Most of the 2026 research is here.

Why agents need in-run routing

The prompt often doesn’t tell you how hard a task is. SWE-Router shows it formally: an issue that reads like a typo can turn into a multi-module refactor. So the cheap model explores for a few turns, and a small predictor (Qwen2.5-Coder-7B with LoRA) reads those turns and predicts whether the cheap model will finish. If not, the strong model restarts from scratch, on purpose, because handing it the cheap model’s reasoning tended to pull it into the same mistakes.

Why bother? On SWE-bench Verified, gemini-3-pro-preview resolved 70.4% of 500 issues for $391. deepseek-v3.2 resolved 62.4% for $22.60. That’s about 17x the spend for 8 points. Reading two exploration turns raised their routing score from 0.627 (prompt-only) to 0.780.

More results: CodeRescue lets a router choose between reflecting, replanning, or escalating after a cheap model fails, and one setting beat always-escalate at 35% of its recovery cost. AgentRouter reports a 12M-parameter classifier assigning each step to one of four model tiers for 72% lower cost at 97.3% of frontier quality. These are single-team results on narrow tests. Treat them as direction.

The hidden cost: switching breaks the cache

A prompt cache belongs to one model. Microsoft’s study of GitHub Copilot traces from June 2026 (13.5M sessions, 761M model calls) found cache hit rates around 90% inside a turn, 55% across a turn boundary, and 8% after a model switch. The median call carried 68K prompt tokens and produced 247 output tokens, so reusing input is most of the bill. About 6.4% of sessions switched models anyway, mostly after errors or rate limits.

Nhu Hoang measured it on a 20-turn session at Claude list rates ($5 per million input tokens frontier, $1 efficient, cache reads 0.1x, writes 1.25x). Moving half the turns to the 5x cheaper model should have cut input cost about 40%. Alternating every turn saved 13%. Switching only at two phase boundaries saved 29%. Running the frontier model with caching broken cost 4.88x the cache-friendly baseline, a bigger swing than any model choice.

So a switch has a price you can compute in advance: tokens the new model hasn’t cached, times its input or cache-write rate. Vercel’s AI Gateway keeps one provider pin per model per session (up to eight), so switching back can land on a warm cache. vLLM Semantic Router’s session-aware mode cut model switches 79% across 21,600 simulated turns.

Be skeptical

LLMRouterBench (400K instances, 33 models) re-ran ten routing methods under one setup and found many performed about the same, and several, commercial routers included, didn’t reliably beat a simple baseline. Judge any router on whole runs, with failed attempts and restarts counted, against three references: all-cheap, all-strong, and random.

An analogy: switching doctors mid-treatment

Seeing a general practitioner for routine care and a specialist for hard cases is smart. But if you switch doctors every visit, each new one has to read your whole file from scratch, and you pay for that reading. Switch at natural moments (a new diagnosis, a referral), not every Tuesday.

Try this

Use the Copilot numbers for a quick calculation. A call carries 68,000 input tokens. At $5 per million, full price is $0.34; a cache read at 0.1x is $0.034. Now imagine switching to a model that’s 5x cheaper ($1 per million) with a cold cache, paying the 1.25x write rate: 68,000 x $1.25 per million is $0.085. Was the cheaper model actually cheaper on that call? (No: $0.085 versus $0.034.) It only pays off if you stay on it for several more calls.

Use it at dinner

“Routing AI requests to cheaper models saves money, but switching mid-task means the new model has to re-read everything. The smart routers only switch at natural breakpoints.”