What Cloud Taught Us About the Real Cost of AI Inference

Updated 6 October 2026. What changed

The headline per-token rate is a poor predictor of enterprise inference spend. One commercial vendor, Inferbase, says a budget built without the rate-limit, caching and side-charge documentation tends to miss its target by 30 to 50% in the first month of production, and gives a worked example in which a $9,000 monthly forecast landed nearer $14,000. This is one vendor's analysis, not an independent benchmark. The mechanics it points to are real, and they are visible in providers' own price lists: caching, output-to-input ratios, rate-limit tiers and side-charges. Finance and FinOps leaders who model inference cost without them are likely to underestimate spend.

The forecast gap

Enterprise teams often find that actual inference bills exceed forecasts built from headline rates. The structural reason is that a rate card shows a base price, while the bill reflects how a workload uses caching, how many tokens it produces, which rate-limit tier it needs and which side-charges apply.

Inferbase, a commercial inference-routing vendor, published a pricing analysis on 13 May 2026 using list prices it captured from provider documentation in the first week of May. Its worked example is an agent with a 30,000-token system prompt making 100,000 requests a day on a $3 per million input-token model. The headline rate predicts about $9,000 a month. Inferbase reports actual bills "in our experience and in conversations with several enterprise teams" closer to $14,000 and sometimes $18,000, which is 55% to 100% above the forecast. It also says a budget built without the rate-limit, cache and side-charge documentation tends to miss by 30 to 50% in the first month of production, and that a fully loaded rate is typically 1.4 to 1.6 times the rate after caching and output-ratio adjustments.

The four cost mechanics rate cards miss

The gap between headline rates and fully loaded cost comes from four mechanics that are largely invisible in provider pricing pages.

1. Caching behaviour

Prompt caching reuses previously processed context at a lower price. Providers' current price lists show cache reads at a small fraction of the base input rate. Anthropic's list, for example, prices cache hits at between 2.5% and 10% of input depending on the model, and charges more than the base rate to write to the cache.

But the saving depends on workload design. If prompts vary slightly between requests, hit rates fall. If cached content expires before it is reused, for example in bursty traffic, the cache is written and not read. Two enterprises using the same model at the same provider can see very different effective costs.

2. Output-to-input token ratios

Providers price output tokens above input tokens. Inferbase puts the output rate at about four to six times the input rate across the main frontier vendors, and Anthropic's current list is consistent with that: $4 input and $20 output per million tokens for Opus 5.5, and $2 and $10 for Sonnet 5.5.

How many output tokens a task produces depends on the task. A summary produces fewer tokens than it reads. Code generation, analysis and agentic loops with tool use and verification can produce many more. A business case that assumes a balanced ratio for a reasoning-heavy workload can understate cost by a large factor, because output dominates the bill.

3. Rate-limit tier pricing

Several providers gate throughput by tier, calibrated to cumulative spend. Inferbase gives the example of an entry-level Anthropic account limited to 30,000 input tokens a minute on one model, enough for roughly one 30,000-token request a minute. A team that points production traffic at it can hit errors on the second concurrent request regardless of willingness to pay. In its words, the first $1,000 or so of spend can be the price of qualifying for a tier where the rest of the spend is possible.

The effect on a budget is friction rather than a surcharge. Pilots run on a low tier and production needs a higher one, which takes time and committed spend.

4. Side-charges and surcharges

The headline rate is a base price. Real deployments accumulate charges that rarely appear in a top-line comparison:

• Region uplift. Anthropic's price list applies a 1.1 multiplier to all token prices for US-only inference on its newer models. Inferbase says OpenAI's regional processing endpoints carry a similar uplift of about 10%.
• Context-tier surcharges. Inferbase reports that some Google Gemini Pro models double their pricing for prompts above 200,000 tokens, while some Anthropic models bill their full context window at the standard rate. Check current terms before relying on either.
• Tool-use fees. Web search, code execution and file search can be billed separately. Inferbase estimates that an agent making several tool calls per turn can see these add 20% to 40% to the token bill. That is the vendor's estimate.
• Retrieval and guardrail costs. Retrieval pipelines and safety checks add infrastructure and per-request cost outside the model invoice.

These charges are real and recurring, and they accumulate until the first full billing cycle reveals the gap.

Reported finding, vendor research. Inferbase worked example, 13 May 2026: an agent forecast at $9,000 a month on headline rates, with actual bills nearer $14,000 and sometimes $18,000. Based on the vendor's experience, not a controlled benchmark.

The 9x spread in list prices

Inferbase, citing Artificial Analysis provider data and provider pricing pages, shows the same open model, Llama 3.3 70B, listed at $0.10 to $0.90 per million input tokens across roughly twenty hosting providers. That is a 9x spread in input list price for identical model weights. It reflects differences in provider cost structure, hardware and throughput. It is a spread in list prices, before caching, output ratios, tiers and side-charges are counted.

For enterprises comparing deployment options, headline comparisons are therefore insufficient, and provider choice is worth more attention than it usually gets.

Self-host crossover economics

For enterprises considering self-hosted infrastructure, the crossover point where owned infrastructure becomes economically competitive with API consumption depends on fully loaded cost, not headline rates.

Inferbase's analysis puts the self-host crossover for a 70B-class model at roughly 10 to 50 billion tokens a month of steady traffic, and suggests not self-hosting below about one billion tokens a month. These are one vendor's estimates and depend on infrastructure choices and on whether side-charges are included in the comparison.

The wide range reflects differences in:

• Infrastructure efficiency and utilization
• Caching implementation quality
• Output-to-input ratios for the workload
• Whether region uplift, context surcharges, and tool-use fees are included in the API cost baseline

Enterprises that compare self-host infrastructure cost to headline API rates will overestimate the crossover point. The correct comparison is fully loaded API cost versus fully loaded self-host cost, including infrastructure, operations, and platform burden. The self-hosting break-even calculator runs that comparison with your own quotes and utilisation.

How to estimate fully loaded cost

CFOs and FinOps leaders should model inference cost using a four-layer framework that captures the mechanics rate cards miss.

Layer 1: Base inference cost

Start with the headline per-token rate for the model, region, and tier. This is the published rate card number.

Layer 2: Caching adjustment

Estimate the share of input that is stable and reusable, and the cache hit rate the workload will actually achieve. Apply the provider's cache-read price to the hit share and the full rate to the rest. Measure hit rates in a pilot rather than assuming them.

Layer 3: Output ratio adjustment

Estimate the output-to-input token ratio for the workload from pilot data. Apply the provider's output price, which is typically several times the input price.

Layer 4: Side-charges

Add region multipliers, context-tier surcharges, tool-use fees, retrieval costs and guardrail overhead, using the provider's current terms and your own architecture.

The result is a fully loaded cost per task. The inference cost estimator runs these four layers with your own rate card and pilot data. Inferbase says its four-step method typically puts the fully loaded rate at 1.4 to 1.6 times the rate after caching and output adjustments.

Governance implications

The forecast gap has direct governance implications for CFOs, CIOs, and FinOps leaders.

For CFOs: AI business cases built on headline rates are likely to be optimistic. Require fully loaded cost modeling before approving scale. Stress-test ROI cases against a materially higher cost, for example 1.5 times the forecast, as a planning sensitivity rather than a measured average.

For CIOs: Architecture and deployment decisions shape fully loaded cost more than provider selection. Caching design, output ratio management, and side-charge minimization are cost governance levers, not just technical optimizations.

For FinOps leaders: Inference cost governance requires workflow-level visibility, not just provider-level reporting. Track cache hit rates, output ratios, tier usage, and side-charge accumulation as operational KPIs.

For engineering leaders: Workload design determines caching efficiency and output ratios. These are not fixed properties of the model. They are design outcomes that engineering controls.

What this means for enterprise buyers

Enterprise AI buyers should treat headline per-token rates as a starting point, not a forecast.

A prudent planning assumption is that fully loaded cost will be higher than headline rates unless the organisation has explicitly modelled the four cost mechanics. Inferbase's own estimate is that the gap is 30 to 50%, which is a useful stress case but not a benchmark.

This does not mean AI is uneconomic. It means that business cases must be built on realistic cost models, not optimistic rate cards.

Enterprises that model fully loaded cost from the start will make better deployment decisions, set better budgets, and avoid the ROI surprises that appear when actual bills arrive.

The lesson from cloud economics applies directly to AI: the invoice is not the rate card. The invoice is what you actually built, how you actually used it, and what you actually paid once all the mechanics are included.

What changed

References and further reading