Most model comparisons are made on the wrong unit. Price per million tokens is easy to find and easy to compare, so it becomes the basis for decisions. But a business does not buy tokens. It buys finished work: a resolved case, a merged change, a correct summary. A cheaper model that fails more often, needs more checking or takes three attempts can cost more per finished piece of work than an expensive one that gets it right first time.
The economic question is not "which model is best?" It is: which setup delivers a successful outcome at the lowest full cost, given the quality the work requires?
The unit that matters
Cost per successful outcome =
(model cost + infrastructure + tool calls + retries + human verification)
÷ share of attempts that clear the quality floor
The quality floor, a term from the Tokenomics Foundation's value classification, is the minimum standard an output must meet before it counts as finished. It is set per workload, before anyone compares models. A legal clause summary and a marketing subject line have very different floors.
Below the floor, a cheaper setup is not cheaper. Failed outputs are paid for twice: once to generate them, and again to fix or redo them.
A worked comparison
Illustrative figures for a document-extraction workload: 100,000 documents a month, with a floor of "all required fields extracted correctly, confirmed by a sample check".
| Setup | Cost per attempt | Clears floor | Human fix per failure | Cost per successful outcome |
|---|---|---|---|---|
| Small model, no check | £0.004 | 78% | £1.50 | about £0.33 |
| Mid-tier model | £0.020 | 92% | £1.50 | about £0.14 |
| Small model, routed to mid-tier when confidence is low | £0.009 | 93% | £1.50 | about £0.11 |
| Frontier model | £0.090 | 96% | £1.50 | about £0.15 |
The method: for each document, the cost of one attempt plus, where it fails, the cost of a person fixing it. For the small model that is £0.004 plus 22% of £1.50. With a person fixing every failure, all four setups eventually produce correct documents; the difference is what it costs to get there.
Three things stand out. The cheapest model per attempt is the most expensive per outcome, because people fix more than a fifth of its work. The frontier model is excellent and still not the best value, because the last few points of quality cost far more than the human fixes they avoid. And the routed setup wins, because it pays for capability only where the work needs it.
Change one assumption and the answer can change. If a person's fix cost £10 rather than £1.50, because errors are found late by a customer, the frontier model's higher clearance rate becomes worth paying for: about £0.49 per outcome against £0.71 for the routed setup. That is why the comparison has to be made per workload.
The options in practice
| Option | When it tends to pay |
|---|---|
| Frontier model | Hard, high-consequence work where errors are expensive or found late. |
| Mid-tier model | The default for most business workloads. |
| Small model | High-volume, well-defined tasks with a clear floor. |
| Routing | Mixed workloads where most items are easy and a minority are hard. |
| Fallback | Reliability: a second provider when the first fails or is slow. |
| Caching | Repeated context, such as long system prompts or shared documents. |
| Batch processing | Work that can wait hours rather than seconds, often at a lower rate. |
| Self-hosted open model | High, steady volume, data constraints, or a need to control versions. |
| On-device | Latency, privacy or offline needs; shifts cost from usage to hardware. |
| Human escalation | Low-confidence items where a person's judgement is cheaper than a stronger model. |
None of these is free. Routing needs evaluation data to set thresholds. Self-hosting moves cost from the provider's invoice to your own engineering and capacity; the self-hosting break-even calculator shows where that pays. Rent, reserve or own covers the sourcing side.
How to run the decision
- Set the quality floor with the business owner, before testing models.
- Build a test set from real work, including the hard cases.
- Measure clearance rate and fix cost for each candidate setup on that set.
- Compute cost per successful outcome at expected volume, including verification. The cost per successful outcome calculator does the arithmetic.
- Pick the cheapest setup above the floor, and record why.
- Re-test when prices or models change. Model choice has a short shelf life.
What would change this view
- If model quality converges so that every option clears most floors, unit price becomes the main lever and this method simplifies.
- If evaluation stays expensive, many organisations will not run step 3 properly, and a default-to-mid-tier policy with routing may be the practical answer.