On MoE decode, attention and experts fail different SLOs and want different GPU pools.

Many of today’s largest models are “mixture of experts” models: instead of one giant brain, they have a crowd of specialist sub-networks and send each word to just a couple of them. That trick saves a lot of compute. It also creates a strange hardware problem, and the fix is to split a single model across two different kinds of machines.

Quick background

When a model writes its answer, it runs a loop called decode: one token at a time. Each step has two main parts:

Attention, where the model looks back over everything said so far. It reads from the KV cache, the model’s running notes on the conversation. Every request has its own notes, so attention is limited by how fast memory can be read (memory-bound).
The feed-forward part (FFN), the heavy math. In a mixture-of-experts (MoE) model, this part is split into many “experts,” and each token only visits a few.

Heavy math is most efficient in big batches: load the weights once, then use them on lots of tokens at the same time.

The problem

MoE sparsity breaks big batches. Take Mixtral 8x22B on an A100 GPU in bf16. It needs about 156 tokens in a batch before the expert math becomes efficient (compute-bound). But with each token going to 2 of 8 experts, each expert only sees about 39 tokens from that batch. Theoretical efficiency for those experts drops to roughly 25%.

Meanwhile attention stays memory-bound no matter what, because every request reads its own notes. Same box, two bottlenecks that refuse to share a sweet spot.

The fix: attention-FFN disaggregation (AFD)

AFD puts attention and experts on separate pools of GPUs.

Attention runs as many copies (replicas), each holding its own conversations’ KV cache. You add more when conversations get longer or more people show up.
Experts run in one shared pool. Tokens from many attention replicas pile into one fat batch per expert, so the math becomes efficient again.

To hide the travel time between pools, the batch is cut into micro-batches that ping-pong back and forth, so one side computes while the other is sending. MegaScale-Infer’s rule of thumb is m ≥ 2(1 + Tc/Tf) micro-batches, usually 3 or 4.

This also lets you mix hardware. MegaScale-Infer’s tables put GPUs that give the most memory per dollar, like H20, on attention, and GPUs that give the most compute per dollar, like L40S, on experts.

The numbers

MegaScale-Infer (ByteDance Seed and Peking University, April 2025): up to 1.90x per-GPU decode throughput versus TensorRT-LLM and vLLM-class baselines, about 1.86x throughput per dollar on H20 plus L40S, and a production deployment on roughly 10,000 GPUs that cut serving cost 1.5 to 2.0x.
Step-3 (StepFun, July 2025), a 321B-parameter model with 38B active: 4,039 tokens per GPU per second on Hopper under a 50 ms per-token target, using a 32-GPU layout. DeepSeek-V3’s published number in the same target band is 2,324, on more GPUs. Their cost model puts Step-3 around $0.055 per million decode tokens at 8K context, versus about $0.068 for DeepSeek-V3 and $0.062 for Qwen3-MoE.

An analogy: the hospital

Picture a hospital where every patient has a personal nurse who knows their chart (attention), plus a few very expensive specialists (experts). If you staff each ward with its own set of specialists, each specialist sees two patients a day and sits idle the rest. Better: keep nurses on every ward, and send patients from all wards to one shared specialist center where each surgeon has a full schedule.

The catch is the hallway. If moving patients takes longer than treating them, you’ve bought traffic jams instead of savings. AFD only works when the two sides are balanced and the trip is short.

Try this

Do the hospital math with the real numbers. A batch of 156 tokens, each going to 2 of 8 experts: 156 x 2 / 8 = 39 tokens per expert. Now imagine 4 attention replicas each sending their 156-token batch to one shared expert pool. How many tokens per expert now? (Answer: 156. Right at the efficient line.) That one multiplication is the whole case for AFD.

Use it at dinner

“In mixture-of-experts models, the ‘remembering’ part and the ‘thinking’ part want different hardware. The newest serving systems put them on different machines and pass tokens back and forth.”