AI lessonsOne weekly lesson on how AI models, serving, and agents work.https://www.nikibirkner.com/Serving agent sessions: keep the KV cache alive across tool and human gaps, route turns back to the warm replica, and price cache TTL by your pause distributionhttps://www.nikibirkner.com/ai-lessons/serving-agent-sessions/https://www.nikibirkner.com/ai-lessons/serving-agent-sessions/AI agents pause constantly for tools and humans. Whether the model's memory survives those pauses decides the bill.Thu, 22 Oct 2026 15:00:00 GMTModel routing and cascades for agents (price the switch; route at phase boundaries; judge on cost per success)https://www.nikibirkner.com/ai-lessons/model-routing/https://www.nikibirkner.com/ai-lessons/model-routing/Sending easy work to cheap models and hard work to strong ones sounds simple. Inside an agent, switching models has a hidden price.Tue, 20 Oct 2026 15:00:00 GMTAgent cost and latency SLOs. Price on cost per success with failures counted; early abort is the biggest margin leverhttps://www.nikibirkner.com/ai-lessons/agent-cost-per-success/https://www.nikibirkner.com/ai-lessons/agent-cost-per-success/The cheapest AI model per token isn't always the cheapest per finished job. And the biggest savings come from giving up early.Sat, 17 Oct 2026 15:00:00 GMTProduction agent monitoring vs offline evals. The offline bank catches known breaks; step monitors, deterministic checks, human reads, and random A/B audits catch drift, and eval-only shipping is earned per change classhttps://www.nikibirkner.com/ai-lessons/production-monitoring/https://www.nikibirkner.com/ai-lessons/production-monitoring/A test suite tells you an AI agent worked on launch day. Here's how teams find out it still works three weeks later.Thu, 15 Oct 2026 15:00:00 GMTScore environment state and process on separate axes; a confident closer is not a DB assert.https://www.nikibirkner.com/ai-lessons/agent-evals-outcome-vs-process/https://www.nikibirkner.com/ai-lessons/agent-evals-outcome-vs-process/AI agents sometimes say 'done!' when nothing happened. Here's how to test whether the work really got done.Tue, 13 Oct 2026 15:00:00 GMTA checkpoint saves state; durable execution also restarts the run.https://www.nikibirkner.com/ai-lessons/durable-execution/https://www.nikibirkner.com/ai-lessons/durable-execution/How long-running AI agents survive crashes, deploys and three-day waits for a human to click approve.Sat, 10 Oct 2026 15:00:00 GMTMulti-agent handoff budgets (1-2k return summary vs shared file store)https://www.nikibirkner.com/ai-lessons/multi-agent-handoffs/https://www.nikibirkner.com/ai-lessons/multi-agent-handoffs/When one AI agent hands work to another, what should it pass along? About two thousand tokens and a pile of file paths.Tue, 06 Oct 2026 15:00:00 GMTCross-session memory: offload beats a longer windowhttps://www.nikibirkner.com/ai-lessons/cross-session-memory/https://www.nikibirkner.com/ai-lessons/cross-session-memory/Every new AI session starts with an empty desk. The trick is leaving good notes for next time.Sat, 03 Oct 2026 15:00:00 GMTAgent loop compaction economics (cache thrash vs prune)https://www.nikibirkner.com/ai-lessons/compaction-economics/https://www.nikibirkner.com/ai-lessons/compaction-economics/Summarizing a long AI conversation to save money can cost more than keeping it all. Here's why.Thu, 01 Oct 2026 15:00:00 GMTInterleaved thinking: effort knobs vs tool-loop input taxhttps://www.nikibirkner.com/ai-lessons/interleaved-thinking/https://www.nikibirkner.com/ai-lessons/interleaved-thinking/When an AI agent thinks between every tool call, yesterday's thinking becomes tomorrow's input, and the bill compounds.Tue, 29 Sep 2026 15:00:00 GMTReasoning decode economics: thinking tokens, relaxed acceptance, SLAshttps://www.nikibirkner.com/ai-lessons/reasoning-decode-economics/https://www.nikibirkner.com/ai-lessons/reasoning-decode-economics/Reasoning models think before they answer. You don't see most of that thinking, but you still pay for it and wait for it.Sat, 26 Sep 2026 15:00:00 GMTNative MTP vs bolt-on EAGLE/Medusa for product ownershiphttps://www.nikibirkner.com/ai-lessons/native-mtp-vs-bolt-on/https://www.nikibirkner.com/ai-lessons/native-mtp-vs-bolt-on/Some models are trained with their own built-in guesser. Others need one bolted on. That choice decides who owns the upkeep.Thu, 24 Sep 2026 15:00:00 GMTPick the speculative proposer from traffic shape and packaging cost; score tokens/sec under your concurrency, not paper acceptance.https://www.nikibirkner.com/ai-lessons/choosing-a-speculative-proposer/https://www.nikibirkner.com/ai-lessons/choosing-a-speculative-proposer/Three ways to guess ahead in speculative decoding, and how to choose between them.Tue, 22 Sep 2026 15:00:00 GMTJeff Dean: Exciting Trends in ML (Rice 2024 watch companion)https://www.nikibirkner.com/ai-lessons/jeff-dean-trends-in-ml/https://www.nikibirkner.com/ai-lessons/jeff-dean-trends-in-ml/A viewing guide to Jeff Dean's 72-minute Rice lecture: scale, specialized chips, the transformer, and what Gemini was built on.Sun, 20 Sep 2026 15:00:00 GMTSplitwise and DistServe show why prefill and decode need separate pools (and often separate GPU SKUs), with concrete cost/throughput gains and when colocation still wins.https://www.nikibirkner.com/ai-lessons/splitwise-distserve/https://www.nikibirkner.com/ai-lessons/splitwise-distserve/Reading the prompt and writing the answer want different machines. Two papers showed how much splitting them saves.Sat, 19 Sep 2026 15:00:00 GMTRead num_key_value_heads before treating GQA as a capacity win.https://www.nikibirkner.com/ai-lessons/gqa-kv-heads/https://www.nikibirkner.com/ai-lessons/gqa-kv-heads/One number on a model card tells you how much memory every conversation costs. Here's how to read it.Thu, 17 Sep 2026 15:00:00 GMTContinuous batching keeps the GPU full; prefix cache only hits if the shared prompt stays identical.https://www.nikibirkner.com/ai-lessons/continuous-batching-prefix-cache/https://www.nikibirkner.com/ai-lessons/continuous-batching-prefix-cache/Two everyday tricks that make AI serving cheap, and the tiny mistake that breaks the second one.Sat, 12 Sep 2026 15:00:00 GMTSell the tier as first claim on capacity, then enforce a fair token budget inside that tier.https://www.nikibirkner.com/ai-lessons/fair-scheduling/https://www.nikibirkner.com/ai-lessons/fair-scheduling/When many customers share one GPU, one customer's burst becomes everyone's slowdown. Here's how fair sharing works.Thu, 10 Sep 2026 15:00:00 GMTBuy GPU capacity in goodput, requests that clear both TTFT and TPOT, not in raw tokens per second.https://www.nikibirkner.com/ai-lessons/goodput/https://www.nikibirkner.com/ai-lessons/goodput/Throughput counts work done. Goodput counts work done fast enough to matter.Tue, 08 Sep 2026 15:00:00 GMTTreat draft length as a live control; a fixed max draft burns compute once the batch is full.https://www.nikibirkner.com/ai-lessons/adaptive-draft-length/https://www.nikibirkner.com/ai-lessons/adaptive-draft-length/Speculative decoding with a fixed setting is tuned for an empty server. The fix is to adjust it on the fly.Sat, 05 Sep 2026 15:00:00 GMTSpeculative decoding pays only when you measure it at real concurrency and on the actual task mix.https://www.nikibirkner.com/ai-lessons/speculative-decoding/https://www.nikibirkner.com/ai-lessons/speculative-decoding/A clever trick that makes models answer faster, and why it can backfire when the servers get busy.Thu, 03 Sep 2026 15:00:00 GMTOn MoE decode, attention and experts fail different SLOs and want different GPU pools.https://www.nikibirkner.com/ai-lessons/attention-ffn-disaggregation/https://www.nikibirkner.com/ai-lessons/attention-ffn-disaggregation/Why the biggest open models are starting to split one model across two kinds of machines.Tue, 01 Sep 2026 15:00:00 GMTIf the GPU is fast and the app still crawls, the missing time is queue, network, or hops, not the model.https://www.nikibirkner.com/ai-lessons/end-to-end-latency/https://www.nikibirkner.com/ai-lessons/end-to-end-latency/On-GPU time versus what the user actually waits through, and where the missing milliseconds hide.Sun, 30 Aug 2026 15:00:00 GMTTTFT is the wait for the first word, TPS is the rest of the stream, and P99 is the speed number to quote.https://www.nikibirkner.com/ai-lessons/ttft-tps-p99/https://www.nikibirkner.com/ai-lessons/ttft-tps-p99/The two numbers behind 'this chatbot feels fast,' and why the average lies.Sat, 29 Aug 2026 15:00:00 GMTStart on a shared token API, and move to dedicated GPUs only when the math or a latency promise forces it.https://www.nikibirkner.com/ai-lessons/shared-vs-dedicated/https://www.nikibirkner.com/ai-lessons/shared-vs-dedicated/Pay-per-token APIs versus running your own GPUs, and the moment the switch makes sense.Thu, 27 Aug 2026 15:00:00 GMTTraining writes the weights; inference reads them on every request, and that is where the cost lives.https://www.nikibirkner.com/ai-lessons/training-vs-inference/https://www.nikibirkner.com/ai-lessons/training-vs-inference/The gym versus game day: why almost every AI product conversation is really about inference.Tue, 25 Aug 2026 15:00:00 GMTHallucination is pattern completion when the truth is not in the context or strongly in the weights.https://www.nikibirkner.com/ai-lessons/hallucination/https://www.nikibirkner.com/ai-lessons/hallucination/Why a model will confidently invent a citation, and what actually reduces it.Sun, 23 Aug 2026 15:00:00 GMTA context window is working memory for one request, not a filing cabinet of past chats.https://www.nikibirkner.com/ai-lessons/context-window/https://www.nikibirkner.com/ai-lessons/context-window/Models read tokens, not words, and they can only see what fits on their desk for this one request.Sat, 22 Aug 2026 15:00:00 GMTNext-token prediction, done extremely well, is how models absorb facts, code, and reasoning.https://www.nikibirkner.com/ai-lessons/next-token-prediction/https://www.nikibirkner.com/ai-lessons/next-token-prediction/Chatbots are not giant search engines. They guess the next word, and getting very good at that turns out to require a lot of knowledge.Sat, 22 Aug 2026 15:00:00 GMT