TTFT is the wait for the first word, TPS is the rest of the stream, and P99 is the speed number to quote.

Two chatbots can be equally “fast” on paper and feel totally different. One makes you stare at a blank box for three seconds and then dumps a wall of text. The other starts typing almost instantly. To talk about speed precisely, you need two numbers and one statistical habit.

Number one: TTFT

TTFT is time to first token. With streaming replies, it’s how long you stare at an empty reply before the first word appears.

TTFT is mostly driven by prefill, the compute-heavy first pass where the model reads your whole prompt before it can say anything. Long prompt, long prefill, longer wait.

Number two: TPS

TPS is tokens per second after that first token. Usually people mean perceived TPS: how fast the stream feels for one user.

Careful, though. The same letters can also mean total TPS across the whole service, which is throughput. If someone brags about “100 TPS,” it’s fair to ask which one.

There’s a cousin called inter-token latency, the gap between tokens. 10 milliseconds between tokens works out to about 100 tokens per second for that user.

Chatbots care about both numbers because the stream is the product. An AI agent making a behind-the-scenes tool call often only cares about total response time, since nobody is reading those tokens one by one.

The statistical habit: quote P99

Inference latency is right-skewed. Most requests bunch near a typical time, then there’s a fat tail of slow ones. The average gets yanked around by that tail and describes almost nobody’s actual experience.

So engineers use percentiles:

• P50 is the median. Half of requests are slower.
• P90 means 1 in 10 is slower.
• P99 means 1 in 100 is slower.

A product that’s snappy most of the time but spikes to several seconds on every tenth reply still feels broken. That’s why inference engineers chase P90 and P99.

An analogy: the coffee shop

TTFT is how long you wait before the barista even acknowledges you. TPS is how fast they make the drink once they start. A shop that greets you instantly and then takes ten minutes feels different from one that ignores you for two minutes and then hands you a cup in thirty seconds.

And P99? That’s the morning the shop took 25 minutes. You don’t remember the average. You remember that morning. So does every customer, which is why the slow tail is the number to watch.

Try this

Grab a stopwatch (your phone has one) and test a chatbot five times with the same medium-length question.

1. Time from pressing send to the first word appearing. That’s your TTFT.
2. Time from the first word to the last word, and roughly count the words. Divide words by seconds for a rough words-per-second (tokens per second will be a bit higher, since a token is about three quarters of a word).
3. Write down all five TTFTs. Look at the slowest one versus the typical one.

Even with five tries you’ll probably see one outlier. Now imagine millions of requests, and you understand why “average latency” is mostly hope.

Use it at dinner

“TTFT is how long until the first word; TPS is how fast the rest streams. Lead with P99 when you talk speed, because the average is mostly hope.”

Sources

• Philip Kiely, Inference Engineering (Baseten Books, 2026), section 1.4. (Book, no link.)