Next-token prediction, done extremely well, is how models absorb facts, code, and reasoning.
The Mon/Wed/Fri briefing is AI news.
Here is a slightly deflating fact about ChatGPT, Claude, Gemini and the rest. None of them is looking things up in a giant encyclopedia when you ask a question. Each one is doing something that sounds almost silly: given the words so far, guess the next one. Then do it again. And again, a few hundred times, until there’s an answer on your screen.
The surprise is how far that one skill goes.
The idea
A language model is a next-token predictor. (A token is a small chunk of text, often a word or a piece of one. We’ll get to tokens properly in the next lesson.) It reads everything in front of it and produces its best guess for what comes next.
That sounds like autocomplete on your phone, and in a sense it is. The difference is scale and quality. To be really good at “what comes next,” across billions of pages of text, a model has to soak up grammar, facts, writing styles, the shape of working code, and the structure of an argument. If you can reliably finish the sentence “The capital of France is…”, you know a fact. If you can reliably finish a half-written function so it runs, you know something about programming. If you can finish a proof, you have picked up something that looks a lot like reasoning.
So prediction at scale is a kind of understanding. Not consciousness. Real capability.
How it got this good
The leap came in 2017 with the transformer, a model design that can look at the whole context at once and decide which earlier words matter for the next guess. Older designs read text more like a person reading through a keyhole, one word at a time, carrying a fuzzy memory forward. A transformer gets to glance back at everything and weigh it.
Take that design, feed it more data, give it more compute, and you get today’s frontier models. Nothing magic was bolted on. The same “guess the next token” job, done by a much bigger model on much more text, kept getting better at harder things.
An analogy: the world’s most obsessive crossword solver
Imagine someone who has done every crossword ever printed, in every language, plus a few billion fill-in-the-blank puzzles made from textbooks, novels, recipes and code. They never memorized a list of answers. They just got so practiced at “what fits here” that they picked up history, chemistry, slang and a decent sense of how to argue a point, because all of that helps you fill in blanks.
Now hand them a blank that has never existed before: “Explain this error message in my Python script.” They don’t look it up. They fill it in, word by word, using everything that practice taught them. That’s a language model.
The analogy also shows the weak spot. A brilliant crossword solver will happily write something plausible in every square, even when they don’t know. Hold that thought. It’s the subject of the hallucination lesson.
Try this
Open any chatbot and type the start of a sentence with no question at all, like:
The three things every houseplant needs are
Don’t ask anything. Just send it. Notice how the model happily continues the text. Now try a half-written sentence in a style, like the opening line of a fairy tale or a sports commentator’s call, and watch it pick up the voice. You’re seeing the raw skill underneath every “assistant” feature: continue the text in the most likely way.
Bonus round: start a sentence that’s slightly wrong (“The Eiffel Tower, located in Rome,”) and see whether it corrects you or just keeps going.
Use it at dinner
When someone says “AI just copies the internet,” try: “It’s not a database lookup. It’s predicting the next word so well that reasoning, coding and writing fall out of that skill. The interesting debate is where prediction ends and real understanding begins.”
Next up: tokens, context windows, and why “how much it can remember” shapes everything.
Sources
Subscribe
One lesson a week. Unsubscribe anytime.