Token-Based vs Scale-to-Zero vs Scale-to-One: How to Run Your Model Layer

Every application with a model underneath it makes the same decision, usually without noticing. You can pay a provider per token, you can rent a GPU only while there is work in the queue, or you can keep a GPU warm all the time. Those three shapes have different cost curves, different latency behaviour, and different amounts of engineering to stand up.

The choice is not about which model is best. It is about the shape of your traffic. This post covers what each option is, the two questions that select between them, and where the cost curves actually cross.

The three shapes

Token-based. You call a managed endpoint and are billed per token in and per token out. Amazon Bedrock and Microsoft Foundry both run this way, and both serve frontier models and open-weight models side by side. There is no infrastructure to provision, no capacity to plan, and no cost at all when nobody is using the feature.

Scale to zero. You serve an open-weight model yourself on a GPU instance, provisioned when work arrives and released when the queue drains. Between runs your cost is storage alone. The tradeoff is that provisioning takes minutes, not milliseconds.

Scale to one. The same deployment as scale-to-zero, with the autoscaling floor set to one replica instead of zero. Capacity above the floor still scales with demand; the floor itself is billed around the clock. You pay for idle in exchange for never provisioning on the request path.

The last point is worth stating plainly, because it collapses a decision people treat as three-way: scale-to-zero and scale-to-one are the same deployment with a different floor. Moving between them is a change to the scaling policy, not a rebuild.

Two questions select the option

Only two things about your workload matter here.

Is a user waiting on the response? If someone is watching a spinner, provisioning latency lands on them. If the work is scheduled — overnight extraction, bulk classification, backlog reprocessing — provisioning is absorbed by the schedule and nobody experiences it.

Is volume high and steady? Not peak volume. Sustained volume. A feature that spikes for two hours a day and sits idle for twenty-two has low steady volume, whatever its peak looks like.

The cell that trips people up is interactive + high volume, because it is the only one where paying for idle capacity is the right answer. And the reason scale-to-zero cannot serve interactive traffic is worth being concrete about: a cold queue means the first request waits for a GPU to be provisioned and a model to be loaded. That is minutes. It is fine when a scheduler is waiting and unacceptable when a person is.

One constraint overrides the table entirely. If your data cannot leave your account boundary, the managed endpoint is off the table regardless of traffic shape, and the choice reduces to batch → scale-to-zero, interactive → scale-to-one. If that applies to you, our guide to AI inference data residency covers the boundary question in more detail.

Where the cost curves cross

Per-token cost rises linearly with volume. Dedicated capacity is flat. So there is always a crossover, and the only real question is where.

Start with the floor. In us-east-1, a g6e.12xlarge — four L40S GPUs, 192 GB of VRAM, enough to serve a 70B-class model at FP8 with room for KV cache — lists at $10.49/hour, or $7,660/month on demand. On a one-year no-upfront EC2 Instance Savings Plan the same instance is $4,826/month. Both numbers are useful: the committed price is what a steady production workload should actually pay, and the on-demand price is what you pay while you are still deciding.

Now the token side. Blending input and output at a 3:1 ratio, a few representative rates on Bedrock:

Divide the monthly floor by the blended rate and you get the crossover volume. But do it carefully, because there are two different crossovers here and they are an order of magnitude apart.

Crossover one: the same model, delivered differently

Compare paying per token for an open-weight model against serving that same model yourself.

• Llama 3.3 70B at $0.72 blended, against a $4,826/month committed GPU: break-even at 6.7 billion tokens per month. Against the $7,660 on-demand price, 10.6 billion.
• gpt-oss-120b at $0.26 blended: 18.4 billion and 29.2 billion respectively.

Six point seven billion tokens a month is roughly 220 million tokens every single day, forever. Most products never reach that. If you are considering a dedicated GPU to save money on an open-weight model you are already calling per-token, this is the number that should make you check your actual usage first.

Crossover two: a different model entirely

Now compare frontier tokens against a self-served open-weight model:

• Claude Sonnet 4.6 at $6.00 blended, against the same $4,826/month floor: break-even at 804 million tokens per month — about 27 million a day. Against on-demand, 1.28 billion.

That is ten times lower, and it is the comparison most cost arguments are quietly making. It is also the one that is only valid if the open-weight model actually passes your evaluations for the task. You are not changing infrastructure, you are changing models. The infrastructure saving is real; whether the output is equivalent is a separate question that benchmarks answer, not spreadsheets.

The formula

crossover (millions of tokens/month) = monthly floor ÷ blended cost per 1M tokens

What moves it: your input-to-output ratio (output tokens usually cost 4–5× input, so an output-heavy workload crosses over sooner), your cache hit rate on the token side, context length, region, and whether you commit to a savings plan. Measure these on your own traffic. Every number above is a starting point, not an answer.

One thing that does *not* move much: the token price is roughly provider-independent. Llama 3.3 70B is $0.72 per million on Bedrock and $0.71 on Azure Foundry. Shopping providers is not where the savings are.

The cost that never appears on the chart

Crossover math compares two bills. It does not compare two amounts of work.

Token-based can be in production in about a month. Scale-to-one takes roughly three. Scale-to-zero takes about four, because the provisioning and teardown lifecycle is genuinely more machinery than a floor that never moves. That is a quarter or more of engineering time before the first request is served — serving configuration, autoscaling policy, version pinning, capacity planning, and an evaluation harness to prove the self-served model matches what you replaced.

If the crossover math says you save $2,000 a month, weigh that against a quarter of engineering plus ongoing operation. The arithmetic often looks different once the denominator includes people.

How XY Space approaches it

We start clients on tokens and measure. A workload whose volume is not yet known carries no fixed commitment on per-token billing, and the migration path to dedicated capacity stays open once the traffic profile is established. The reverse — standing up GPU infrastructure for a feature whose usage nobody has measured — is the expensive mistake, and it is the common one.

When the volume is real, the decision should be made on measured evidence: candidate models scored on your own samples, cost per unit of work at your actual volume, a latency profile against your traffic shape, and the crossover point calculated for your workload rather than borrowed from a blog post. That is a short, fixed-scope piece of work, and it leaves you with an evaluation harness you keep.

If you are weighing this decision for a workload of your own, talk to us.

Sources

• Amazon EC2 On-Demand Pricing
• Amazon EC2 G6e Instances
• Amazon Bedrock Pricing
• AWS Savings Plans Pricing
• Azure AI Foundry Pricing