Continuous batching keeps the GPU full; prefix cache only hits if the shared prompt stays identical.

Two ideas do a huge amount of the work behind cheap AI answers. One keeps the GPU busy. The other avoids doing the same work twice. The second one can be broken by something as small as putting a timestamp in the wrong place.

Continuous batching

Old-style batching worked like a bus: wait until everyone’s aboard, drive the route, and nobody gets off until the slowest passenger reaches their stop. Short replies sat around waiting for the longest one.

Continuous batching schedules at every step. The engine runs one pass of the model, then rebuilds who’s in the batch. Finished requests leave immediately. Waiting ones hop on.

The Orca paper (OSDI 2022) proved the win on GPT-3 175B. Against FasterTransformer plus Triton at matched latency, throughput went up as much as 36.9x.

vLLM still runs on that idea. Each step, the scheduler hands out a token budget across requests, capped by settings called max_num_batched_tokens and max_num_seqs. Running requests get budget first; waiting ones fill what’s left. Long prompts get chopped into chunks across steps (“chunked prefill”).

That budget is a real product choice. Around 2048, less prompt-reading rides along with answer-writing, so the stream stays steady. At 8192 and above you chew through more prompt per step, which helps first-token time and raw throughput. Chat and big offline jobs want different numbers.

Prefix caching

Many requests start the same way: the same system instructions, the same tool descriptions, the same documents. Prefix caching saves the model’s work on that shared beginning and reuses it.

In vLLM, the saved work is stored in blocks (default 16 tokens each), and each block is identified by a fingerprint (hash) that includes the fingerprint of the block before it, like links in a chain. A later request that matches those exact tokens skips recomputing them. Only full blocks count, so if a shared prompt ends three tokens into a block, those three get no credit.

The tiny mistake

Because each block’s fingerprint depends on everything before it, any difference near the start breaks the whole chain. If a user ID, a timestamp, or a session field sits before the shared system prompt, the chain breaks at block 0 and the cache almost never hits, even though the prompts look identical to a human.

The second trap: a cache lives on one machine. If a load balancer spreads requests randomly across many machines, every cache stays cold. Fixing routing makes a big difference:

SageMaker’s prefix-aware routing on Llama 3.1 70B moved cache hit rate from roughly 25% to over 80% and cut median time-to-first-token by up to 71 to 77%.
Baseten with NVIDIA Dynamo reported about 50% lower time-to-first-token and 34% lower time per token at roughly 89% hit rate across four replicas.

An analogy: the sandwich shop

A sandwich shop pre-toasts bread for the most common order. That’s the prefix cache. But it only helps if your order starts the same way. If you say “Hi, it’s 12:03, I’m Sam, I’d like the usual,” and the shop matches orders word by word from the start, your “usual” never matches anything. Put the personal stuff at the end.

And if the shop has six counters and you’re sent to a random one each time, only one of them toasted your bread. That’s the routing problem.

Try this

If you write prompts for any tool that reuses them (a custom GPT, a saved prompt, an automation), look at the first lines. Is anything there that changes every time, like today’s date, a name, or a greeting? Move it to the end. In apps that use prefix caching, that one edit can mean faster and cheaper replies. Even if you can’t measure it, it’s a great habit.

Use it at dinner

“AI servers save money by reusing work on prompts that start the same way. Put one changing word at the front, like a timestamp, and the savings vanish.”

Continuous batching keeps the GPU full; prefix cache only hits if the shared prompt stays identical.