The decision to reserve or own AI capacity is usually made on a break-even calculation: at a given volume and utilisation, dedicated capacity costs less per token than paying a provider for each request. Rent, Reserve or Own Intelligence? and the self-hosting break-even calculator cover that decision.
Once the capacity exists, the economics change. Its cost is now largely fixed for the term of the reservation or the life of the hardware, while demand from the teams using it varies by hour, by month and by project. The question becomes who pays for it, including for the hours it sits idle. That is an allocation problem, and technology business management (TBM) and IT financial management have been solving versions of it for shared infrastructure for years. AI capacity adds three complications: the unit is expensive, demand is uneven, and the alternative of calling an external model is always one configuration change away.
In short: put the full cost of the shared capacity in one pool each month, including the platform team and serving software. Allocate it with a driver each team can see and influence: reserved capacity to the team that reserved it, metered use by GPU-hours or tokens, and spare capacity at a discount to batch work. Keep headroom and idle capacity visible as their own lines rather than spreading them across users, because spreading them raises the internal price and pushes teams towards external APIs. Start with showback and move to chargeback when the numbers are trusted.
Key takeaways
- Shared AI capacity has a mostly fixed cost and variable demand. Allocation decides whether teams see a price that reflects what they use and what they reserve.
- The pool should include the people and software that run the platform, not just GPUs.
- Reserved capacity is charged to whoever reserved it, whether used or not. Metered use is charged by a driver such as GPU-hours or tokens.
- Idle capacity has three causes, headroom by policy, reservations not used, and over-provisioning, and each has a different owner.
- Spreading idle cost across users raises the internal rate. If that rate passes the price of an equivalent external model, teams move out and utilisation falls further.
- Kubernetes cost allocation already handles idle cluster cost the same way. GPUs make the same problem sharper: the unit is coarser and dearer, scales down more slowly, and has an external substitute.
- FOCUS 1.3 added columns for disclosing how shared costs are split, which gives the allocation method a standard place in cost data.
What goes in the pool
The pool is everything it costs to provide the capacity for a month. For self-hosted or reserved AI capacity that usually includes:
- Compute. Reserved GPU instances, or the monthly depreciation of owned hardware, with hosting, power and cooling where they are charged separately.
- The serving stack. Inference servers, model gateway, vector stores, monitoring and any licences.
- The platform team. The engineers who run, patch, scale and support the service. This is often the largest cost left out.
- Shared overheads. Security reviews, evaluation runs and model updates that serve every tenant.
The Components of AI TCO has the full list. The point here is that the rate charged to teams should be built from the whole pool. A rate based on GPU cost alone undercharges, and the shortfall appears later as an unexplained central cost.
Four ways to split the pool
| Method | How it works | Where it fits | What it encourages |
|---|---|---|---|
| By reservation | A team books dedicated capacity, such as a whole GPU or a fixed share, and pays for it whether used or not | Production services with steady demand or latency guarantees | Careful sizing, because unused reservation is the team's cost |
| By metered use | Each team pays for what it consumes, measured in GPU-hours, tokens or requests | Variable or exploratory workloads on shared capacity | Efficient use, but also a temptation to move work elsewhere if the rate looks high |
| By fixed share | Each team pays an agreed percentage, set at the start of the year | Early platforms where metering is not yet reliable | Simplicity, but little incentive to use capacity well |
| Hybrid | A reserved base plus metered use above it, with spare capacity sold at a discount | Mature platforms with a mix of production and batch work | The closest match between cost and behaviour, at the price of more metering |
Most organisations end up with a hybrid. The choice of driver matters more than the method. A driver works when each team can see it, influence it, and recognise it as a fair reflection of what it uses. GPU-hours suit reserved and long-running work. Tokens or requests suit shared inference endpoints where many teams send traffic to the same model.
Fractional sharing makes metering more precise. NVIDIA's Multi-Instance GPU can partition a single GPU into as many as seven isolated instances, each with its own memory and compute, so several light inference workloads can share one card and each be measured separately.
Idle capacity is a decision, not a rounding error
Every shared platform has hours when capacity is not doing productive work. The temptation is to spread that cost across the teams that did use it. The better approach is to split idle capacity by cause, because each cause has a different owner.
- Headroom held by policy. Capacity kept free so a service can meet its latency or availability target at peak. Whoever set the target owns this cost: usually the service owner, sometimes a central budget where the organisation has decided the resilience is shared.
- Reserved but unused. A team booked capacity it did not use. This is the team's cost, and it is the signal that the reservation is too large.
- Unallocated. Capacity that nobody reserved and nobody used, often because the platform was sized for demand that has not arrived. This belongs to the platform owner, and it should stay visible as its own line.
That last line is the reason to keep idle cost visible. A rate inflated by capacity nobody used tells teams the platform is expensive. If the inflated rate passes the cost of an equivalent external model, the rational response is to route work to the external model, which lowers utilisation and raises the rate again.
The KPI library tracks this line as Unallocated Shared AI Capacity Cost.
How idle GPU cost compares with Kubernetes idle cost
FinOps teams that already allocate Kubernetes cluster costs will recognise the shape of this problem. The OpenCost specification, the open standard from the Cloud Native Computing Foundation project of the same name, splits a cluster's cost into workload costs and cluster idle cost. Each workload is charged the greater of what it requested and what it used. Idle cost is whatever the cluster's assets cost that no workload claimed. Overhead such as cluster management fees is a further line, and idle and overhead can be kept separate or distributed across tenants by a stated method. That is the identity this article uses: pooled cost equals allocated cost plus unallocated cost.
Three Kubernetes habits carry straight across. Charging the greater of request and use is the same rule as charging a reservation whether it is used or not, so a team that books more than it needs sees the cost. Keeping idle as its own line, and choosing deliberately whether to spread it, is the same discipline as the three idle causes above. And labelling every namespace and workload with an owner, covered under tagging below, is what makes either possible.
The differences are why GPU idle cost needs more attention than CPU idle cost usually gets.
| Factor | General-purpose Kubernetes cluster | Shared GPU platform |
|---|---|---|
| Unit of allocation | Fractions of a CPU core and memory, so many small workloads pack onto one node | Whole devices by default. A workload that needs part of a GPU holds all of it unless partitioning or time-slicing is set up |
| Cost of an idle unit | Low per hour, so idle is often tolerated as a rounding error | High per hour, so the idle line is material and needs an owner |
| Scaling down | Stateless services start in seconds, so autoscaling recovers much of the idle capacity | Loading model weights takes time, so teams keep capacity warm. That is headroom by policy, and autoscaling recovers less of it |
| Alternative supply | Few direct substitutes for running the workload in the cluster | An external model API is often one configuration change away, so an inflated internal rate drives work out |
If Kubernetes cost allocation is already in place, extend it rather than build a separate model for AI. Add GPU-hours as an allocated resource, split GPU idle by the three causes above rather than leaving it as one cluster-wide figure, and add the platform team, serving software and model updates to the pool, because cluster cost tools see the infrastructure but not the people and licences that run the service.
Internal pricing tiers
A single internal rate treats a guaranteed reservation and a job that can wait until tonight as the same product. They are not. Three tiers cover most needs.
- Reserved. Guaranteed capacity at the fully loaded rate, charged whether used or not. For production services with latency targets.
- Metered. Shared capacity at the fully loaded rate, charged for what is used, with no guarantee at peak. For variable and development work.
- Batch. Spare capacity at a discount, which the platform can pause when reserved or metered demand needs it. For evaluation runs, document processing and other work that can wait.
The external price for equivalent quality is the test for all three. If the metered rate is persistently above the price of calling a comparable external model, the platform is too large for its demand, or its costs are too high, and the sourcing decision should be revisited. Inference choices and AI cost and the inference cost estimator give the external comparison.
Showback first, chargeback when the numbers are trusted
Showback reports each team's share without moving money. Chargeback moves it into the team's budget. For a new AI platform, showback is usually the right start: it lets teams see and question the numbers before they affect a budget, and it gives the platform time to fix metering.
Move to chargeback when four conditions hold:
- The driver is measured reliably, and teams can reconcile their share to their own usage data.
- The pool reconciles to the source invoices or the fixed asset register every month.
- Teams have the controls to change their use: reservation sizes, model choice, batch scheduling.
- The internal rate has been compared with the external alternative, and the gap is understood.
Chargeback without those conditions moves arguments about the numbers into budget negotiations, where they are harder to settle. The federated operating model sets out who should own each part.
Quotas, priority and tagging
Allocation needs metering, and metering needs every workload to carry an owner. The FinOps Foundation's guidance on AI and machine learning workloads on Kubernetes recommends allocating cost through namespaces and labels so that every workload lands in a tagged bucket for showback and chargeback, tracking idle against billable time, and using per-team resource quotas and budget alerts to stop excess GPU requests before they are scheduled.
Quotas and priority classes are also how the tiers above are enforced: reserved work gets guaranteed quota, metered work shares what remains, and batch work runs at the lowest priority and is pre-empted first.
Reporting the split in FOCUS
The FinOps Open Cost and Usage Specification (FOCUS) added split cost allocation in version 1.3, released in December 2025. It gives data providers columns to disclose how a shared resource's cost has been divided across workloads, including the allocation method, the method's details and the resource the cost was allocated to. That means the method behind a team's share can travel with the number, rather than living in a spreadsheet only the platform team understands.
For an internally run platform, the same idea applies even if the data never leaves the organisation: record the pool, the driver, each share and the unallocated remainder, and keep the method next to the figures.
From GPU-hours to cost per outcome
A fair internal rate is the start, not the end. A team that pays its share knows what its capacity costs. It still needs to know what that capacity produced. The step from cost per GPU-hour to cost per outcome is the same for internal capacity as for external APIs: divide the team's allocated cost, plus the review and rework the AI creates, by the outcomes that clear the quality floor. Where capacity is the constraint, return on scarce capacity is the better guide to which workloads should get it.
Questions readers ask
How should GPU costs be allocated across teams?
Build one monthly pool that includes compute, serving software and the platform team. Charge reserved capacity in full to the team that reserved it. Charge shared use by a metered driver, usually GPU-hours for long-running work and tokens or requests for shared inference endpoints. Offer spare capacity at a discount to batch work. Show headroom and idle capacity as separate lines with named owners.
Who should pay for idle GPU capacity?
It depends on why it was idle. Headroom kept for a service's peak belongs to the service owner, or to a central budget if the organisation has chosen to share resilience. A reservation a team did not use belongs to that team. Capacity nobody reserved belongs to the platform owner and should stay visible, because it is the signal to resize the platform or bring work in.
Is idle GPU cost the same as Kubernetes idle cost?
The accounting is the same. In both, pooled cost equals the cost allocated to workloads plus an idle remainder, and workloads are charged the greater of what they requested and what they used. The economics differ: GPUs are allocated as whole devices by default, cost far more per idle hour, scale down more slowly because models take time to load, and have an external API as a substitute. So the idle line matters more and needs a named owner.
Should AI platforms use showback or chargeback?
Start with showback while metering and reconciliation settle. Move to chargeback when the driver is reliable, the pool reconciles to invoices or assets each month, teams can control their usage, and the internal rate has been compared with external prices.
What utilisation should a shared GPU platform target?
There is no single right number, because headroom for latency and availability is a legitimate use of capacity. Track utilisation alongside response times, as the GPU and AI compute utilisation KPI suggests, and watch the unallocated line: if it is persistently large, the platform is bigger than its demand.
Tools and templates
Self-hosting break-even calculator
Browser tool. Compare paying per token with running an open-weight model on dedicated GPUs, including utilisation and people. Runs in your browser; nothing is sent anywhere.
Fully loaded inference cost estimator
Browser tool. Build a monthly inference estimate from the rate card up, adding output ratio, caching, uplifts and fees. Runs in your browser; nothing is sent anywhere.
AI Portfolio and Cost Register
Excel template. One register of initiatives, tools and agents with owners, and one cost register in the seven layers of the AI TCO Framework.
Sources: FinOps Foundation, FOCUS 1.3 press release (11 December 2025) and FOCUS split cost allocation documentation; FinOps Foundation, "Scaling Kubernetes for AI/ML Workloads with FinOps" (30 September 2025); NVIDIA, Multi-Instance GPU (read 26 October 2026); OpenCost Specification v01 and Kubernetes documentation, Schedule GPUs (both read 26 October 2026).