Cross-session memory: offload beats a longer window

Inside one long conversation, an AI can ride a cache and keep everything in view (see the compaction lesson). But open a new session tomorrow and it starts with an empty desk. The question for anyone building with AI agents isn’t “how big can the window get?” It’s “what do we write down so the next session can pick up where this one left off?”

Three layers of memory

1. In-window (keep-all). You resend a stable beginning (instructions, tools, history) and pay cache-read rates while the bytes stay identical. On Anthropic most models charge about 0.1x input for cache reads. This only helps within one continuous conversation.

2. Instruction and file memory, loaded at the start. Claude Code reads files like CLAUDE.md or AGENTS.md, optional rule files scoped to certain paths, plus “auto memory.” Auto memory loads the first 200 lines or 25KB of MEMORY.md, whichever comes first, and pulls topic files on demand. Aim for under about 200 lines per instruction file so the model actually follows it; files over 4 MiB are skipped. Cursor Rules and AGENTS.md do the same job from the editor side. These files are standing orders and an index, not a full transcript.

3. Tool memory, on demand. Anthropic’s memory tool runs on your side. The model can view, create, edit, insert, delete or rename files under /memories, and your app owns the storage. The API also nudges a habit: check memory before working, and write progress as if the window could reset any moment.

Anthropic’s long-running agent pattern goes further. A first “initializer” session writes a setup script, a progress log, a feature checklist (often JSON with passes: false), and a first git commit. Later sessions read those, run a quick test, ship one feature, commit, and update the log.

Why this matters

Across sessions, the failures are different from mid-session ones. A fresh session might glance at half-finished work and declare the project done. Or try to finish everything in one go and die halfway. Good notes (progress, open bugs, decisions) matter more than the whole chat.

What the numbers say

Anthropic’s internal agentic-search eval: context editing alone +29% over baseline; editing plus the memory tool +39%.
Letta’s August 2025 test on the LoCoMo memory benchmark: a plain filesystem plus file-search tools hit 74.0% with gpt-4o-mini, above Mem0’s then-cited 68.5%. Letta also disputed how one competitor was scored in that comparison.
Mem0’s later vendor suite (2026) claims about 92.5% on LoCoMo at roughly 7k tokens per query, while full-context runs can burn 25k+ tokens for the same question. In Mem0’s head-to-head table, OpenAI Memory and LangMem scored about 53% and 58%.

Treat vendor leaderboards as directional. Retrieval quality and how well the agent uses its tools often matter more than which memory product you pick.

On the consumer side, OpenAI reports about 5x lower compute to serve Free users with its 2026 “Dreaming” memory rollout, which grew saved memories into background synthesis.

The rule of thumb

Keep-all for the active session. Offload across sessions. Prefer a short, always-loaded index (a CLAUDE.md, a MEMORY.md, or a progress log) and pull details on demand. Reach for a fancy vector or graph memory service when many users share long-lived facts and your agent already retrieves well. Re-embedding a whole transcript every turn is usually worse than a short progress file plus a git log.

One safety note: if an agent reads untrusted content (like web pages), think twice before letting it write to memory other sessions will trust. Anthropic’s docs call out a read-only option for that case.

An analogy: the night-shift handoff

Hospital shifts don’t hand over by replaying the last twelve hours of video. The outgoing nurse writes a short handoff note: what happened, what’s pending, what to watch. The incoming nurse reads it in two minutes and gets to work. A longer video wouldn’t help. A better note would.

Try this

Next time you use any AI assistant for a multi-day project, end each session by asking it: “Write a 10-line handoff note for whoever continues this tomorrow: what we decided, what’s done, what’s next, open questions.” Save it. Tomorrow, start a fresh chat by pasting only that note. Compare how quickly you get back to work versus scrolling through yesterday’s chat.

Use it at dinner

“The trick to AI memory isn’t a bigger window. It’s a good handoff note, the way hospital shifts do it.”