Start on a shared token API, and move to dedicated GPUs only when the math or a latency promise forces it.

If you’ve ever built anything with an AI model, you probably started by grabbing an API key and paying per token. At some point every growing AI app asks the next question: should we keep renting by the token, or get our own machines? The answer is a lot like a familiar money question in ordinary life.

Two ways to serve a model

Shared inference means you send traffic to a public API and pay per million tokens. You get an endpoint, an API key, and someone else’s GPUs.

The upside: the model is already warm and running, and there’s very little engineering for you to do. It’s perfect while you’re still figuring out whether anyone wants your product. The downside: the bill grows with every token, their uptime is your uptime, and a noisy neighbor (another customer flooding the same service) can steal your speed. Rate limits and which model versions are available are decided on their side of the fence.

Dedicated deployments mean you rent (or buy) GPUs and run a serving setup that exists only for your app. You pay for GPU hours even when traffic is quiet. In return, you pick the model, the latency target, and who else shares the machine (nobody).

When the switch happens

In Philip Kiely’s framing, the move usually happens once the math or the product forces it:

• Volume. Traffic gets high enough that paying for a GPU by the hour is cheaper than paying the token meter.
• Customization. You need a fine-tuned model, or a speed promise a public API won’t make.
• Pipelines. Later on, multi-step workflows push you to keep several models sitting close together.

An analogy: taxis versus owning a car

When you rarely drive, taxis win easily. No insurance, no parking, no maintenance, pay only per ride. As you start driving every day, the per-ride cost adds up, and at some point owning a car is cheaper even though it sits in the driveway at night. You also get to choose the car. But owning means you handle the oil changes.

Shared APIs are taxis. Dedicated GPUs are owning the car. Neither is “better.” It depends how much you drive.

A common mix-up

Shared versus dedicated is about how a model is served. Open versus closed is about whether the weights are public. These are different questions.

Plenty of open models run on shared pay-per-token endpoints. Some closed labs will sell you a dedicated setup. What open weights do give you is the option to run the full dedicated path yourself, on whatever hardware you choose.

Try this

Do the taxi-versus-car math on a made-up app. Pick a pay-per-token price from any public AI provider’s pricing page, and a GPU hourly rental price from any cloud provider’s pricing page. Then estimate:

1. How many tokens your app uses per day (try 10 million, then 1 billion).
2. The daily cost at the token price.
3. The daily cost of one GPU running 24 hours.

You don’t need to know how many tokens a GPU can serve to see the shape: at small volume the token meter wins by a mile, and at large volume the fixed GPU cost starts to look reasonable. That crossing point is the whole decision.

Use it at dinner

“Start on a shared meter. Move to dedicated when the GPU-hour math wins, or when you need a custom model or a latency promise the public API can’t make.”

One more wrinkle worth knowing: the same model can need two different setups. A chat app wants a fast first word. A nightly batch job wants cheap throughput and doesn’t care about speed.

Sources

• Philip Kiely, Inference Engineering (Baseten Books, 2026), section 1.1. (Book, no link.)