Splitwise and DistServe show why prefill and decode need separate pools (and often separate GPU SKUs), with concrete cost/throughput gains and when colocation still wins.

Every AI reply happens in two phases: first the model reads your prompt, then it writes the answer one token at a time. For years both phases ran on the same GPU. Two influential papers, Splitwise from Microsoft Research (ISCA 2024) and DistServe (OSDI 2024), made the case for giving each phase its own machines. Now it’s a standard feature of production serving.

The two phases want different things

Prefill eats the whole prompt in parallel, builds the KV cache (the model’s notes), and produces the first token. Once the prompt is long enough, it’s limited by raw compute.
Decode walks token by token, reading those growing notes every step. It’s limited by memory speed and memory size.

Newer GPUs widened the gap. Splitwise’s comparison of H100 versus A100: about 3.43x the compute, only about 1.64x the memory bandwidth, and the same 80 GB of memory. Each generation helps prefill much more than decode.

On a shared GPU, a long prefill in the batch stretches every decode step, so everyone’s stream stutters. Tricks like chunked prefill soften this, but both phases still share one schedule and one memory, so teams over-buy hardware to protect both speed targets.

The split

Splitwise puts prompt machines in one pool and token machines in another. After prefill, the notes move over a fast network (InfiniBand, or NVLink inside a server). Their optimized path ships notes layer by layer while the next layer is still computing, so visible transfer stays under about 7% of prompt time and about 0.8% of end-to-end time on their coding-trace test.

DistServe does the same split, then tunes the GPU count and parallelism for each phase separately against first-token and per-token targets. They report up to 7.4x higher request rate or 12.6x tighter targets than combined setups, while keeping more than 90% of requests within latency.

Different GPUs for different jobs

Splitwise-HA runs H100 for prefill and A100 for decode. They report up to 1.4x higher throughput at 20% lower cost, or 2.35x more throughput under the same power and cost budgets. Decode on A100 lost less than prefill did relative to H100 (about 0.70x versus about 0.51 to 0.54x on their Llama-70B test). Capping decode GPUs’ power by about 50% barely moved token speed.

Production systems now do this routinely. NVIDIA Dynamo decides whether to prefill remotely or locally based on queue depth and prompt length. Mooncake (Moonshot, the team behind Kimi) separates prefill and decode clusters plus a shared pool of notes in regular memory and SSD; their writeups report handling on the order of 100 billion-plus tokens per day and roughly doubling request capacity versus their earlier stack.

When combining still wins

Splitting pays when per-token speed targets are strict, prompts are long or bursty, traffic is high, and the network is fast. Keeping both phases together still wins on a few GPUs, light traffic, short prompts where the transfer is a real chunk of first-token time, and batch jobs where you just want every GPU full. Uneven traffic can also leave one pool idle while the other is swamped. Philip Kiely’s rule of thumb in Inference Engineering: reach for disaggregation at high daily token volume, large models (around 100B parameters and up), and prompt-heavy traffic.

An analogy: the bakery

A bakery has two jobs: mixing dough (heavy, powerful, bursty) and decorating cakes (slow, careful, one at a time). Put both at one counter and every time a giant mixing job starts, the decorators have to stop. Give them separate rooms, and the mixing room can have the industrial mixer while decorating gets cheaper tables and more people. Just make sure the hallway between the rooms is short.

Try this

Notice the two phases yourself. Paste a very long document into a chatbot and ask for a one-line summary. Then ask a short question that needs a long answer (“write a 500-word story about a lighthouse”). The first one makes you wait before any words appear (heavy prefill, short decode). The second starts quickly and then streams for a while (light prefill, long decode). You just felt why these want different machines.

Use it at dinner

“Reading your prompt and writing the answer are such different jobs that big AI services now run them on different machines, sometimes even different chips.”