Buy GPU capacity in goodput, requests that clear both TTFT and TPOT, not in raw tokens per second.

Here’s a puzzle. A team upgrades its AI serving setup, and the dashboard shows tokens per second going up. Users say it got worse. Both are true. The missing idea is goodput, a term engineer Ammar Ahmad has been pushing as the right way to think about inference.

The two halves of a request (thirty-second version)

One request hits the model.

Prefill reads the whole prompt, fills the KV cache (the model’s notes on the conversation), and emits the first token. That wait is TTFT, time to first token.
Decode then walks token by token, rereading those notes each step. The time between tokens is TPOT (time per output token), also called ITL (inter-token latency).

Prefill hammers the chip’s compute in short bursts. Decode likes big batches and is limited by memory speed. When both share one pool of GPUs, a new long prompt makes every in-progress reply pause.

Throughput versus goodput

Throughput counts completed work. Goodput counts completed work that also met both speed targets: first token fast enough, and the stream fast enough.

DistServe, from the Hao AI Lab (OSDI 2024), defines it as the maximum request rate where at least 90% of requests clear both budgets. That’s why tokens per second can climb while users still wait on the first word.

A worked example from DistServe

In their simple walkthrough of a 13B model on A100 GPUs:

Running prefill and decode together, goodput lands around 1.6 requests per second per GPU. That’s the lower of the two limits: how many requests you can serve before TTFT breaks, versus before TPOT breaks.
Splitting into two prefill GPUs and one decode GPU lands around 3.3 requests per second per GPU.

Moving the conversation notes from the prefill machine to the decode machine sounds expensive, but for OPT-175B at 2048 tokens over eight PCIe 5.0 links it takes about 17.6 ms, less than one decode step.

Headline results: up to 4.48x goodput versus vLLM on summarization (or a 10.2x tighter speed target). The paper abstract reports up to 7.4x more requests or 12.6x tighter targets while keeping more than 90% within latency.

The 2026 update

The vLLM MORI-IO post (April 2026) saw the same pattern on one 8-GPU MI300X node serving Qwen3-235B-A22B-FP8. At 8 requests per second, a split setup cleared 73 of 100 requests on both targets. The standard combined setup cleared 26 to 30. The spiky gaps between tokens vanished. The remaining misses were first-token time: splitting buys smooth streaming and costs a longer path to the first word.

An analogy: the restaurant that counts plates

A restaurant brags that it serves 500 plates a night. But if 200 of them arrived cold, after the customer gave up, that’s not 500 meals. Goodput is counting only plates that came out hot and on time. Separating the prep station from the grill (prefill from decode) is how busy kitchens stop one giant order from freezing every other ticket.

When splitting doesn’t win

Splitting isn’t always right. Keeping prefill and decode together still wins for short outputs, weak network links between machines, or chatbots where first-token time is the main pain and traffic is light. Ray Serve’s docs say the same.

Try this

Make your own goodput score for any app you use. Over a day, note ten times you asked a chatbot something. For each, mark two boxes: “first word came fast” and “rest came fast.” Your throughput is 10. Your goodput is the number of times both boxes are checked. If it’s 6, that app’s experience is really a 60% product, whatever its dashboard says.

Use it at dinner

“Throughput counts every reply. Goodput only counts replies that were fast to start and fast to finish, and that’s the one users feel.”