A context window is working memory for one request, not a filing cabinet of past chats.

Last lesson: models guess the next word. This one is about what they can actually hold in their head while they guess. It explains a lot of “wait, why did it forget that?” moments.

Tokens first

Models don’t see letters or words. They see tokens, little chunks of text. “cat” is usually one token. “unhappiness” might be two or three. A smiley or a rare name can eat several.

Rough English math: 1,000 tokens is about 750 words. When someone says a prompt was “128k,” they mean 128,000 tokens, not pages.

The desk

The context window is the model’s working desk for this one request. Everything has to fit on it:

your question
the chat so far
any files you attached
the hidden instructions the app gives the model before you ever type (the “system prompt”)

The desk is not long-term memory, and it is not what the model “learned in training.” Training is the gym. The context window is RAM. If it isn’t on the desk right now, the model cannot see it.

That’s why “the AI remembered that from last month” is usually not what happened. Either the old message is still sitting on the desk in this same conversation, or some other system (saved notes, a database, a “memory” feature) quietly put it back on the desk before your question went in. When the desk fills up, older stuff falls off the front, or the request just fails.

An analogy: the substitute teacher

Picture a very smart substitute teacher who walks into a classroom with zero memory of yesterday. Whatever is written on the whiteboard and in the notes left on the desk, they can use brilliantly. Whatever isn’t there, they simply don’t know, no matter how many times the regular teacher covered it.

Every request to a model is a new substitute walking in. The app’s job is to leave the right notes on the desk. The bigger the desk, the more notes fit, but a giant pile of notes is also harder to search.

Big windows, and the catch

Frontier models in 2026 commonly advertise around a million tokens of context. That’s a few books, or a mid-size codebase.

The expert tell is advertised versus usable. Stuffing the window is expensive, because you pay for every token on the desk each time. And recall in the middle of a giant dump often gets worse: the model is sharper on the beginning and end of what you gave it than on page 400 of 800. That’s why RAG (retrieval-augmented generation: fetch the few relevant passages first, then ask) still exists even in the era of huge windows.

Try this

Paste a long article into a chatbot (a few thousand words is fine).
Pick a small, specific detail from the very middle, like a number or a name.
Ask about it.
Then start a new chat and ask the same question without pasting anything.

In the first chat the answer was on the desk. In the second, the model either says it doesn’t know or makes something up. Same model, same question, different desk. If you want to go further, paste the article plus a second long article and see whether middle details get fuzzier.

Use it at dinner

“A million-token window sounds like infinite memory, but it isn’t. It’s RAM, not a filing cabinet. And the model still gets fuzzy in the middle of a giant dump, so smart teams retrieve the relevant bits instead of pasting the whole company wiki.”

Sources

Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)