Module 2: What AI really costs

Watch: the module in 1:15

Captions are on by default and can be turned off in the player. The full transcript is below.

Read the transcript

The model invoice is the cost everyone sees. It is rarely the cost that decides whether AI pays.

The AI TCO framework sets out seven layers: infrastructure, data, models, integration, people, governance and operations. The costs most often missing from a business case sit in the last four.

In one worked example, a support agent costs twenty-five pence a ticket on the invoice. Fully loaded, with review, integration and people, it costs about three pounds.

Even the token bill hides four mechanics: caching that may not hit, output tokens priced several times above input, rate-limit tiers, and side-charges for tools and retrieval.

Then there is the inference choice. On pay-per-token, the provider carries the idle hours. On dedicated capacity, you do. Self-hosting pays only when volume is high, steady and measured.

In the exercise, a headline cost of £840 becomes nearly £1,400 fully loaded. Find which mechanic adds the most.

By the end of this module you can

Name the seven layers of AI TCO and the costs most often left out of a business case.
Count each cost once across SaaS, APIs, agents and owned infrastructure.
Build a fully loaded inference cost from a rate card: caching, output ratio, uplifts and fees.
Judge when self-hosting is likely to pay, and which inputs decide it.

Finance and FinOps lenses: the cost denominator in every later case comes from this module.

The argument in brief

A business case usually prices what is easiest to price: the API rate card, the seat licence or the hosting bill. Those figures are real, but they describe one layer of a seven-layer cost structure. Review and rework, integration, data work, governance and a fair share of the shared platform sit in other budgets, which is why they go missing.

This module builds the cost side of every decision in the course. The AI TCO Framework gives the seven layers. The Components of AI TCO gives the rule that stops double counting: count the purchased service or the underlying resources, never both. The two inference pieces show why the bill is not the rate card, and why the cost per token of a dedicated deployment is set mainly by how busy the hardware is kept.

Key ideas

The seven layers: infrastructure, data and context, models, integration and workflow redesign, people and capability, governance and compliance, operations and portfolio oversight.
A managed API or SaaS fee already pays for the provider's servers and power. Do not add them again.
Fully loaded inference cost adds caching, the output premium, uplifts and per-request fees to the headline rate.
Halving the price of one cost line does not halve total cost. Cost per successful outcome is the decision measure.
Self-hosting pays when volume is high and steady, utilisation is measured, and the team can run it. Compare three options: a proprietary API, an open model on a serverless host, and the open model on dedicated capacity.

Terms to know

AI TCO: The total cost of owning and operating an AI capability across model access, infrastructure, data, governance, integration, and support.
Total cost of AI (TCA): In the Tokenomics Foundation's value classification, the costs that cannot be written in the unit of the baseline, such as tokens, GPU hours, power, licences and the labour to build and operate the platform.
Inference Cost: The recurring cost of generating outputs from a deployed AI system in production.
Reasoning Tokens: Tokens a reasoning model generates internally before it gives its visible answer. They are billed at the output rate, usually not shown to the user, and often several times longer than the answer itself.
Model Hosting: The way an AI model is deployed and run in production, whether through a managed provider, dedicated service, or self-hosted infrastructure.
Model Routing: The logic used to direct different requests to different models, providers, or configurations based on cost, quality, latency, or policy.

Read

In this order. Each opens on the main site.

Worked example: when does a dedicated endpoint beat serverless?

Illustrative figures. One dedicated endpoint, two GPUs at £2.50 an hour each, running all month (730 hours), costs £3,650. A modest setup can serve 10 billion tokens a month at full load; a well-tuned setup on an input-heavy workload can serve 30 billion. The serverless comparison is an open model at about £0.60 per million tokens.

Steps of the worked example
20% utilised	Modest setup £1.83 per million tokens; well-tuned £0.61. The modest setup costs about three times the serverless price.
50% utilised	Modest £0.73; well-tuned £0.24, on compute alone.
Add people	Half an engineer at £4,000 a month adds about £0.27 per million tokens to the well-tuned setup at 50% (15 billion tokens), taking it to £0.51.
80% utilised	Well-tuned setup with the engineer included: about £0.32 per million tokens.

The result

At 50% the self-hosted route is barely cheaper than serverless once its staff are counted, with more operational risk. At 80% the saving is clear. Utilisation and the fixed cost of people decide it, not the GPU price.

Self-hosting pays when volume is high, steady and measured. Price the idle hours and the people before the hardware.

From How Inference Choices Change AI Cost, where the figures are explained in full.

Try it: from rate card to fully loaded cost, then the self-hosting test

A document assistant makes 100,000 requests a month, each with 3,000 input tokens and 500 output tokens. The model costs £2.40 per million input tokens and £12 per million output tokens. 60% of the input is a stable system prompt and shared policy text, with a 70% cache hit rate, cache reads at 10% of the input price and a 25% write premium. Regional processing adds 10% to token cost, and retrieval and guardrail checks cost £0.002 a request.

Figures for the exercise
Requests a month	100,000
Input and output tokens per request	3,000 and 500
Input and output price per million	£2.40 and £12
Reusable input and cache hit rate	60% and 70%
Cache read price and write premium	10% and 25%
Uplift and fees	10% and £0.002 a request

Open the estimator with the figures above, or enter them yourself.
Note the headline cost (every token at the input price) and the fully loaded monthly cost.
Work out the ratio between them, and which single mechanic adds the most.
Then open the self-hosting calculator: 12 billion tokens a month, £0.60 per million on a serverless host, two GPUs at £2.50 an hour per copy, 30 billion tokens of capacity per copy, 80% maximum utilisation and £4,000 a month of people. Is self-hosting cheaper at this volume?

Open the inference cost estimator with these figures(opens in a new tab)

Show the worked answer

Headline cost is £840 a month. Fully loaded cost is about £1,388 a month, about £0.014 a request and 1.65 times the headline.

The output premium adds the most (£480), because output is priced at five times input. Caching saves about £240, the 10% uplift adds about £108 and fees add £200.

Self-hosting: one copy costs £3,650 a month in GPUs plus £4,000 of people, £7,650 in all, against £7,200 on the serverless host. At 12 billion tokens the endpoint runs at 40% utilisation and the API is cheaper. Break-even is about 12.75 billion tokens a month, and the people cost is what keeps it there.

The rate card is a starting point. Output ratio, fees and utilisation decide the bill.

Check your understanding

Five practice questions. The final exam uses different questions in the same style.

Progress is saved in this browser only.