If the GPU is fast and the app still crawls, the missing time is queue, network, or hops, not the model.

Last lesson split chat speed into time-to-first-token and tokens-per-second. This one is a quieter trap that catches experienced teams: which clock are you actually reading?

Two clocks

On-GPU time (also called inference-only time) is how long the chip spends generating tokens.

End-to-end time adds everything else between the user’s phone and that chip: waiting in a queue, network hops across the internet, and any middleware services the request passes through on the way.

Philip Kiely keeps both numbers in his book Inference Engineering because they answer different questions. Inference-only time tells you whether your model work helped. End-to-end time tells you what the human actually waits through.

A worked example

Say your GPU dashboard reports 80 ms time-to-first-token. Users still say the first bubble takes about half a second.

That leaves roughly 420 ms unaccounted for. It usually lives in one of three places:

a busy queue where the request waits its turn,
a far-away region, so the request crosses a continent and back,
three service hops before the request even reaches the accelerator.

Here’s the punchline: a fancier GPU kernel won’t shrink that gap. Speeding up the math on the chip does nothing about a request sitting in line or flying across the ocean. Shorter paths and more capacity will.

An analogy: the fast chef in a slow restaurant

The chef plates your dish in 80 seconds flat. You still waited eight minutes. Why? You stood at the host stand, the server took your order to the wrong kitchen, and the ticket sat on a rail behind twelve others.

If you only time the chef, you’ll keep buying faster stoves. If you time the whole meal, you’ll fix the host stand.

Try this

You can see the “path” part of latency yourself.

Pick any website or app and load it on home Wi-Fi.
Load it again on mobile data, or while connected to a VPN set to another continent.
Notice the difference in how long it takes before anything appears.

The server didn’t get slower. The trip got longer. AI apps work the same way, with extra stops for queues and middleware. If you’re technical, open your browser’s developer tools, go to the Network tab, and look at the timing breakdown for a request: you’ll see “waiting” and “connection” chunks that have nothing to do with the server’s actual work.

Use it at dinner

“If the GPU is snappy and the app still crawls, chase the path to the user before you retune the model.”

Sources

Philip Kiely, Inference Engineering (Baseten Books, 2026), section 1.4.2. (Book, no link.)