Sell the tier as first claim on capacity, then enforce a fair token budget inside that tier.

When lots of customers share the same pool of GPUs, one customer’s sudden burst of traffic becomes everyone else’s slow reply. It’s the AI version of one roommate running every faucet in the house while you’re in the shower. This lesson is about how serving platforms keep things fair.

How bad it gets

An open-source project called kvwarden measured this on an A100 GPU running Llama-3.1-8B on vLLM.

A quiet customer alone on the system saw worst-case (P99) time-to-first-token of 53.9 ms.
With a “flooder” sharing the same first-come-first-served engine, that climbed to 1,585 ms, about 29x worse.
With a per-customer token bucket at the front door, the quiet customer’s P99 landed at 61.5 ms, just 1.14x their solo number.

The engine itself never knew who the customers were. A proxy in front of it did. Fairness happened at the door, before requests joined the batch.

Why “requests per minute” isn’t enough

Sheng et al.’s Virtual Token Counter (VTC) paper points out that equal request rates still starve short jobs. If one customer sends 2,000-token prompts and another sends 200-token turns, “one request each” isn’t equal at all. Fairness has to count tokens in and out, even though you can’t know the output length when a request arrives. VTC keeps a running virtual token count per customer and proves that busy customers stay within a 2x service difference of each other, without leaving GPUs idle.

How a production system layers it

Cohere described a four-layer setup in June 2026, applied in a fixed order:

Rate limiter: admit or reject.
Performance tier: honor the paid service level.
Deficit Round Robin: inside a tier, rotate turns across customers, so a customer with a 10,000-request burst can’t bury one with 5 requests.
Per-customer queue: once a customer gets its turn, its own requests go in order of urgency and deadline.

They use request-based budgets for chat-style streaming endpoints and token-based budgets for embedding and reranking batches, because “one request” is a terrible unit when prompt sizes differ by 100x.

The research debate

Strict fairness can hurt efficiency. Requests with the same prompt prefix can reuse cached work, but a fairness scheduler that ignores that leaves speed on the table. Deficit Longest Prefix Match (Cao et al., January 2025) keeps a fairness guarantee while recovering up to 2.87x throughput versus VTC, and cutting per-customer latency up to 7.18x versus a strong distributed baseline. Equinox (August 2025) reports up to 1.3x throughput, 60% lower time-to-first-token and 13% higher fairness than VTC at 94% GPU utilization.

And should you keep separate machines for chat and for batch jobs? Microsoft Research’s Niyama (March 2025) mixes them on one fleet with fine-grained priorities: +32% serving capacity versus separate fleets, and under extreme load an order-of-magnitude drop in missed targets by setting aside the roughly 5% of requests already doomed so the rest stay fast. When a customer is over budget, an honest “try again in a moment” (HTTP 429 with Retry-After) beats a silent 10-second wait.

An analogy: the airline boarding line

Airlines sell priority boarding. First class gets first claim on the jet bridge. That’s the tier. But inside economy, you still board by group, so one family of twelve doesn’t block the whole plane, and each family boards its own kids in whatever order it likes. Tier first, then fair turns inside the tier, then each customer’s own priorities.

Try this

Next time you’re at a grocery store, watch the lanes. The “10 items or fewer” lane is a token-based budget: it’s not about how many customers, it’s about how much work each brings. Then think about a restaurant that only limits “parties per hour.” Why does a party of 2 and a party of 20 count the same? You just rediscovered the VTC paper’s main point.

Use it at dinner

“Shared AI servers have a roommate problem. The fix is to sell priority as who goes first, then split time fairly by how much work each customer brings, not how many requests.”