Niki Birkner

Founding PM at Together AI, working on inference and growth. Co-founder of Babel, a community for people building AI in San Francisco.

Latest

Serving agent sessions: keep the KV cache alive across tool and human gaps, route turns back to the warm replica, and price cache TTL by your pause distribution

Frontier

AI agents pause constantly for tools and humans. Whether the model's memory survives those pauses decides the bill.

Read the lesson

The Mon/Wed/Fri briefing is AI news.

Get it by email

Start here

A path through the ideas. After these, follow the dates.

  1. 01 Next-token prediction, done extremely well, is how models absorb facts, code, and reasoning. Foundations
  2. 02 A context window is working memory for one request, not a filing cabinet of past chats. Foundations
  3. 03 Hallucination is pattern completion when the truth is not in the context or strongly in the weights. Foundations
  4. 04 Training writes the weights; inference reads them on every request, and that is where the cost lives. Foundations
  5. 05 Start on a shared token API, and move to dedicated GPUs only when the math or a latency promise forces it. Foundations
  6. 06 TTFT is the wait for the first word, TPS is the rest of the stream, and P99 is the speed number to quote. Foundations
  7. 07 If the GPU is fast and the app still crawls, the missing time is queue, network, or hops, not the model. Foundations
  8. 08 Jeff Dean: Exciting Trends in ML (Rice 2024 watch companion) Foundations
  9. 09 Buy GPU capacity in goodput, requests that clear both TTFT and TPOT, not in raw tokens per second. Systems
  10. 10 Continuous batching keeps the GPU full; prefix cache only hits if the shared prompt stays identical. Systems
  11. 11 Read num_key_value_heads before treating GQA as a capacity win. Systems
  12. 12 Reasoning decode economics: thinking tokens, relaxed acceptance, SLAs Frontier
  13. 13 Speculative decoding pays only when you measure it at real concurrency and on the actual task mix. Systems

All lessons

Foundations 8

Jeff Dean: Exciting Trends in ML (Rice 2024 watch companion) If the GPU is fast and the app still crawls, the missing time is queue, network, or hops, not the model. TTFT is the wait for the first word, TPS is the rest of the stream, and P99 is the speed number to quote. Start on a shared token API, and move to dedicated GPUs only when the math or a latency promise forces it. Training writes the weights; inference reads them on every request, and that is where the cost lives. Hallucination is pattern completion when the truth is not in the context or strongly in the weights. A context window is working memory for one request, not a filing cabinet of past chats. Next-token prediction, done extremely well, is how models absorb facts, code, and reasoning.

Systems 10

Score environment state and process on separate axes; a confident closer is not a DB assert. A checkpoint saves state; durable execution also restarts the run. Cross-session memory: offload beats a longer window Agent loop compaction economics (cache thrash vs prune) Splitwise and DistServe show why prefill and decode need separate pools (and often separate GPU SKUs), with concrete cost/throughput gains and when colocation still wins. Read num_key_value_heads before treating GQA as a capacity win. Continuous batching keeps the GPU full; prefix cache only hits if the shared prompt stays identical. Sell the tier as first claim on capacity, then enforce a fair token budget inside that tier. Buy GPU capacity in goodput, requests that clear both TTFT and TPOT, not in raw tokens per second. Speculative decoding pays only when you measure it at real concurrency and on the actual task mix.

Frontier 11

Serving agent sessions: keep the KV cache alive across tool and human gaps, route turns back to the warm replica, and price cache TTL by your pause distribution Model routing and cascades for agents (price the switch; route at phase boundaries; judge on cost per success) Agent cost and latency SLOs. Price on cost per success with failures counted; early abort is the biggest margin lever Production agent monitoring vs offline evals. The offline bank catches known breaks; step monitors, deterministic checks, human reads, and random A/B audits catch drift, and eval-only shipping is earned per change class Multi-agent handoff budgets (1-2k return summary vs shared file store) Interleaved thinking: effort knobs vs tool-loop input tax Reasoning decode economics: thinking tokens, relaxed acceptance, SLAs Native MTP vs bolt-on EAGLE/Medusa for product ownership Pick the speculative proposer from traffic shape and packaging cost; score tokens/sec under your concurrency, not paper acceptance. Treat draft length as a live control; a fixed max draft burns compute once the batch is full. On MoE decode, attention and experts fail different SLOs and want different GPU pools.

↑↓ to move · enter to open · esc to close · t toggles theme