How Inference Choices Change AI Cost: A Guide for Budget Owners

Two teams can run the same open model, on the same hardware, for the same workload, and one can pay several times more per token than the other. The difference is not the model. It is a set of serving decisions that are usually made by engineers, rarely written down, and almost never reviewed by the person who owns the budget.

This guide explains those decisions in budget terms. It does not ask you to configure anything. It aims to help you recognise which choices move cost, ask the right questions of your team or supplier, and judge when self-hosting is worth the effort.

The one number behind all of this

Every self-hosted or dedicated deployment comes down to one ratio:

Cost per million tokens = cost of the hardware for the period
                        ÷ millions of tokens actually served in that period

The numerator is mostly fixed: a GPU costs the same per hour whether it is busy or idle. The denominator depends on two things: how many tokens the setup can produce per hour, and how much of that capacity is actually used, its utilisation.

That is why the choices in this guide matter. Some raise capacity (batching, serving engines, quantisation). Some change utilisation, or move the risk of low utilisation to someone else (serverless versus dedicated). Under a pay-per-token API, the provider manages this ratio and builds it into the price. Under a dedicated endpoint, you manage it, and your finance team carries the result.

Throughput versus latency

A GPU serving a language model is most efficient when it handles many requests at once. Modern servers group requests in flight into a batch and process them together, so the cost of each pass through the model is shared. Larger batches mean more tokens per GPU-hour and a lower cost per token.

The trade-off is speed for each user. In a larger batch, each request waits a little longer for its first word and receives later words a little more slowly. Engineers describe this as latency (how long one person waits) against throughput (how much total work the hardware does).

The useful question is how fast each workload actually needs to be:

Pay-per-token providers price this trade-off directly. Several major API providers, including OpenAI and Anthropic, offer batch processing at about half the standard rate for jobs returned within 24 hours. If a workload can wait and is not using a batch option, it is likely paying a speed premium it does not need.

Serverless versus dedicated

There are two main ways to pay for inference on open or custom models, and they allocate risk differently.

Serverless (pay per token) means the provider runs shared infrastructure and charges for the tokens you send and receive. You pay nothing when idle. The provider carries the utilisation risk and prices for it, and you are limited to the models in their catalogue, their rate limits and their latency on a busy day.

Dedicated (pay per GPU-hour) means you rent specific GPUs, either from a cloud provider or a specialist inference platform, and run a model on them. You pay for every hour the endpoint is up, whether it serves one request or a million. In return you can run any model, including a fine-tuned one, with predictable performance and no shared rate limits.

There are positions in between. Some platforms scale dedicated endpoints down to zero when idle, at the cost of a delay of seconds to minutes while a model reloads. Committed or reserved capacity lowers the hourly rate in exchange for a term. Rent, reserve or own covers those sourcing options in depth.

A common and avoidable pattern is to move to dedicated hosting for a predicted volume, then run the endpoint at low utilisation because demand arrives later, in bursts, or not at all. The hourly bill is fixed; the tokens served are not. Before committing, ask what utilisation the business case assumes and how it will be measured.

The specialist GPU clouds and inference platforms that sell dedicated capacity are supply-side infrastructure. The vendor map refers to them as context rather than profiling them, because it focuses on the tools buyers use to see and control spend.

Open-weight models

An open-weight model is one whose trained weights are published, so anyone can host it. The main families in enterprise use include Meta's Llama, Alibaba's Qwen, DeepSeek's models and Mistral's open releases.

Open weights change the economics in three ways.

1. No model-maker margin on each token. You pay for hosting, not for the model itself. On a serverless host, open models are often priced well below proprietary models of similar size.
2. Hosts compete on price for the same model. Because many providers can serve identical weights, prices vary widely. One analysis found Llama 3.3 70B listed at $0.10 to $0.90 per million input tokens across about twenty hosts; The Real Cost of AI Inference sets out that figure and its caveats.
3. The model does not change under you. A self-hosted or pinned open model behaves the same until you choose to upgrade, which helps with evaluation, audit and regulated use.

These gains come with obligations that do not appear on an invoice:

• Evaluation. An open model is cheaper only if it clears the quality floor for the workload. The comparison should be on cost per successful outcome, as set out in The Economics of Model Selection, not on price per token.
• Licence review. "Open" does not mean the same thing across families. Alibaba released its dense Qwen3 models under Apache 2.0, and DeepSeek released R1 under the MIT licence. Llama uses Meta's own community licence, which requires organisations whose products had more than 700 million monthly active users at a version's release to request a separate licence from Meta, and carries an acceptable use policy. Mistral publishes some models under Apache 2.0 and others under commercial terms. Licences differ between versions, so legal review should be of the specific model and version.
• Provenance and policy. Some organisations restrict models by origin, training data disclosure or jurisdiction. These are policy questions to settle before engineering effort is spent.
• Upgrades. No one upgrades a self-hosted model for you. Each new version needs evaluation, testing and redeployment.

Serving engines

A serving engine is the software that loads a model onto GPUs and handles incoming requests. The engine decides how requests are batched, how memory is used and how efficiently the hardware is kept busy. The same model on the same GPUs can produce materially different output per hour depending on the engine and how it is tuned.

The most widely used options include:

• vLLM, an open-source engine from UC Berkeley research. Its authors reported throughput two to four times higher than the leading serving systems of the time at the same latency, mainly through more efficient memory management (Kwon et al., SOSP 2023).
• TensorRT-LLM, NVIDIA's engine, which compiles models for specific NVIDIA hardware. It can extract more performance on that hardware but generally needs more engineering effort to set up and maintain.
• SGLang and others, which are also used in production and compete on similar techniques.

Benchmarks between engines move with each release and depend heavily on the model, hardware and workload. A budget owner does not need to choose an engine. The right expectation is that the team can state, and has measured, the tokens per GPU-hour their setup produces for the actual workload, and has tested at least one alternative configuration.

On a pay-per-token API, engine efficiency is the provider's problem and is built into the price. On a dedicated endpoint, an untuned engine is a direct cost. If capacity is half what a tuned setup would deliver, the effective cost per token doubles.

Quantisation

Quantisation stores a model's weights at lower numerical precision, for example 8 bits or 4 bits per weight instead of 16. The model takes less memory and moves less data per token, so it can run on fewer GPUs and serve more requests at once.

The memory effect is easy to see. A 70-billion-parameter model at 16 bits needs about 140 GB for its weights alone, which is more than one 80 GB data-centre GPU, so it needs at least two. At 8 bits it needs about 70 GB; at 4 bits, about 35 GB, leaving room on a single GPU for the working memory that serving many users requires. Fewer GPUs per copy of the model means a lower hourly bill for the same capacity.

The question is what is lost. A large study of the Llama 3.1 family, covering more than 500,000 evaluations, found 8-bit floating point (FP8) quantisation effectively lossless across model sizes, 8-bit integer quantisation with a 1 to 3% accuracy drop when well tuned, and 4-bit weight-only quantisation competitive with 8-bit integer formats (Kurtic et al., 2024, published at ACL 2025).

Many serverless hosts already serve quantised versions of open models, and that is part of why their prices are low. It is worth asking a provider which precision they serve, because two hosts listing the same model may not be serving the same thing.

When self-hosting pays off

Self-hosting here means running an open-weight model on dedicated GPUs you rent or own, rather than paying a provider per token. Its cost per token is set by the ratio at the start of this guide, plus the cost of the people who run it.

The illustrative table above as curves. Compute only; staff cost comes on top.

To run the same comparison with your own quotes and measurements, use the self-hosting break-even calculator.

The figures are invented to show the shape of the decision, not to predict any real deployment. The shape is the point: self-hosting can be much cheaper or much dearer than serverless, and the swing comes from utilisation and tuning, not from the hourly GPU rate. GPU rental prices themselves vary several-fold between providers and moved upwards for some contract terms in 2026 as capacity tightened, so the hourly figure should be checked against current quotes.

The pay-per-token side moves too. In October 2026 one provider launched a small proprietary model at $0.10 per million input tokens and $0.50 per million output tokens (Anthropic pricing). If a model of that size meets the quality bar for the workload, the volume at which self-hosting pays off rises sharply, so compare against the cheapest model that clears the bar, not only against the model you would self-host.

Published estimates of the crossover point exist but should be used as scenarios. One vendor's analysis puts it for a 70B-class model at roughly 10 to 50 billion tokens a month of steady traffic and advises against self-hosting below about one billion a month, as discussed in The Real Cost of AI Inference.

Self-hosting tends to pay when most of these are true:

• Volume is high and steady, so dedicated capacity can be kept busy across the day and the week, not only at peak.
• Utilisation is measured from real traffic, not inferred from a forecast.
• The model is not available on a serverless host, for example because it is fine-tuned on your own data.
• Data, residency or sovereignty requirements rule out shared services.
• Predictable latency is worth paying for, and shared rate limits have caused real problems.
• The team can run it, including on-call support, security patching, model upgrades and capacity planning.

It tends not to pay when demand is spiky, early-stage or uncertain, when no one owns the platform, or when the main motive is a per-token comparison against a proprietary API that ignores the serverless open-model option.

Questions for your team or supplier

These questions can be asked without technical knowledge, and the answers reveal whether inference cost is being managed or simply incurred.

1. What response time does each workload need, and which workloads could run in batch at a lower rate?
2. Are we paying per token or per GPU-hour for each workload, and why that model?
3. For dedicated endpoints, what was utilisation last month, measured from traffic rather than estimated?
4. How many tokens per GPU-hour does our setup produce on the real workload, and when was that last tested against an alternative configuration?
5. Which precision are we running, and was the quantised model tested against the workload's quality floor?
6. Have we tested an open-weight model on a serverless host for this workload before considering self-hosting?
7. Who has reviewed the model licence, for the specific version in use?
8. What would it cost to switch host or model, and what lock-in are we accepting?

For finance and FinOps readers, most of these choices have a cloud equivalent:

For FinOps and ITFM teams, the first, third and fourth questions translate into regular measures: cost per million tokens by endpoint, utilisation by endpoint, and tokens per GPU-hour. They sit alongside cost per inference and cost per successful outcome in the AI Economics KPIs.

Sources and further reading

• Kwon, W. et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023. Source of the vLLM throughput comparison.
• Kurtic, E. et al., "Give Me BF16 or Give Me Death? Accuracy-Performance Trade-Offs in LLM Quantization", 2024, published at ACL 2025. Source of the quantisation findings.
• OpenAI, "Batch API", and Anthropic, "Batch processing". Provider documentation on discounted batch rates; check current terms.
• Meta, "Llama 4 community licence"; Qwen, "Qwen3 release"; DeepSeek, "DeepSeek-R1 repository". Licence terms as published by the model makers; terms differ between versions.