Training writes the weights; inference reads them on every request, and that is where the cost lives.

Most AI headlines are about training: a new model, a giant cluster, a record-setting run. But the moment you actually use ChatGPT or Claude, you’re touching the other half of the story, and that other half is where most of the money and most of the product headaches live.

The idea: the gym and game day

Training is the gym. You show the model billions of examples and nudge its internal numbers (the weights) every time it guesses wrong. Slowly those numbers come to encode language, then code, then a lot of the public internet. A frontier training run can take months and a warehouse full of GPUs. When it finishes, the weights freeze.

Inference is game day. Someone types a prompt. You load those frozen weights and decode tokens, one after another, to produce a reply. Every chatbot answer you’ve ever read was inference. Philip Kiely’s book Inference Engineering defines the field simply: serving generative models in production.

Why the split matters

Training is rare and capital-heavy. Big labs do their largest runs a handful of times a year.

Inference is the meter that keeps spinning. Every user, every message, every token. When people complain that a chatbot is slow at 4pm, or a startup worries about its AI bill, or an engineer talks about “dollars per million tokens” and uptime, those are inference problems.

This also explains a common mistake. If one answer comes back wrong and someone suggests retraining the model, they’re proposing a gym rebuild for a game-day issue. Usually the right fix is cheaper and faster: improve the prompt, give the model better retrieved documents, or fix the serving setup.

Why generative inference became its own field

Older machine learning inference was a small job. Think of a fraud score or a product recommendation: one quick calculation on an ordinary CPU, one number out.

Generative models are different because they produce a sequence of tokens, and each extra token costs time and memory. A 500-word answer is hundreds of steps, each one reading through the model’s weights. That’s why specialized serving engines like vLLM, SGLang and TensorRT-LLM exist. Their whole job is making game day fast and cheap.

An analogy: writing the cookbook versus running the restaurant

Training is writing a cookbook. It takes years of testing, it’s expensive, and when it’s done, it’s done. Inference is running a restaurant that cooks from that book every night. The cookbook was a one-time cost. The kitchen staff, the gas bill, the line out the door at 7pm and the complaint that table 12 waited 40 minutes are forever. If a dish comes out wrong, you don’t rewrite the cookbook first. You check the kitchen.

Try this

Next time you use a chatbot, watch the reply arrive and count roughly how long it takes to write a long answer (ask for “a 400-word explanation of photosynthesis”). Then ask for a one-sentence answer to the same question. The long one took longer because every token is another step of inference. Now multiply that by the millions of people asking questions at the same moment, and you can see where the cost lives.

For a second experiment: ask the model a question about something that happened last week. If it doesn’t know, that’s the frozen-weights part of the story. Game day can’t change what was learned in the gym.

Use it at dinner

“Training writes the weights. Inference reads them, over and over, for every user. Most of the money and the product pain now live on the read side.”

Want to go deeper? A single request has two halves, called prefill and decode, and you can even run them on different machines. That shows up in later lessons.

Sources

• Philip Kiely, Inference Engineering (Baseten Books, 2026). (Book, no link.)
• vLLM documentation