Module 4: Measuring AI value

Watch: the module in 1:12

Captions are on by default and can be turned off in the player. The full transcript is below.

Read the transcript

Hours saved multiplied by salary is not a return. Most of the work comes before any money is mentioned.

A defensible claim starts with a workload, a baseline and a quality floor. Observe the change. Net out the review and rework the AI created. Say what happened to the time released. Only then convert it to money and book it.

Outputs that fail the quality floor are cost, not value: rejected drafts, reverted code, customers who come back.

Then grade the evidence, from E0, an anecdote, to E4, controlled and independently validated. The rule is simple: no monetised claim below E2.

Underneath, cost data knows about API keys and outcome data knows about cases. Only a workload identifier carried by both can join them.

In the exercise, an insurer's four and a half million pounds of AI spend turns out to be only about a quarter outcome-linked. Find what holds coverage down, and why the denominator matters.

By the end of this module you can

Build a value claim with a baseline, a quality floor, netting and a stated conversion.
Grade evidence from E0 to E4 and apply the rule that no monetised claim sits below E2.
Specify the data and the workload identifier that join cost to outcome.
Compute attribution coverage and state which test and denominator it uses.

Everyone, and especially the technology lens: workload identity is a platform decision.

The argument in brief

Value measurement usually fails in one place. Someone reports that a tool saved time, multiplies the hours by a salary and calls the result a return. That skips most of the work.

A defensible claim starts with a workload, a baseline and a quality floor. It nets out the review and rework the AI created. It says what happened to the time released, and only then converts the change into money. The Tokenomics Foundation's value classification calls these steps Observe, Net, Convert and Book. This site adds two things at the point where a claim meets a decision: an evidence level, and a decision with a review date.

The module also covers the plumbing. Cost data knows about API keys. Outcome data knows about cases and orders. Only a workload identifier carried by both can join them, which is why attribution coverage depends on it.

Key ideas

A value claim is not a return. A return also needs the full cost, on the same workload boundary.
Outputs that fail the quality floor are cost, not value: rejected drafts, reverted code, repeat contacts.
Released time becomes money only if it is redeployed to valued work or the cost leaves a budget. Say which.
Evidence levels: E0 anecdote, E1 before and after, E2 matched cohort, E3 holdout or quasi-experiment, E4 controlled and independently validated. No monetised claim below E2.
Attribution coverage divides by all identified AI spend, not by the spend already allocated.

Terms to know

AI Baseline: The measured pre-AI state of the process, metric, or outcome against which AI-related improvement is claimed.
Quality floor: What makes an AI output count as finished at all, together with the rate at which outputs clear it.
Observe, Net, Convert, Book: The four-step claim chain in the Tokenomics Foundation's value classification.
Counterfactual evidence level: A five-step scale, E0 to E4, for how strong the evidence is that an outcome was caused by AI.
Value claim: A defensible statement about the incremental benefit of an outcome, measured or estimated against a baseline.
AI value ledger: A small operating record for a material AI workflow that treats value evidence like financial evidence.
Cost per Outcome: The cost of achieving a defined business result such as a resolved case, converted lead, or completed internal workflow.

Read

In this order. Each opens on the main site.

Worked example: a customer-service agent, netted to the quality floor

Illustrative figures. A retail contact centre handles 200,000 contacts a month.

Steps of the worked example
Baseline	Cost per resolved contact £4.20; repeat-contact rate 14%.
Quality floor	A contact counts as resolved only if there is no repeat contact on the same issue within seven days.
Observe	The agent fully handles 38% of contacts. Of those, 11% generate a repeat contact.
Net	About 67,600 contacts a month resolved by the agent and clearing the floor.
Convert and book	Avoided handling cost at £4.20, less the agent's £0.55 per contact, booked as cost avoidance because the centre has stopped backfilling leavers.
Evidence and full cost	E2, from a region-by-region rollout. Platform, integration, quality monitoring and one analyst: about £52,000 a month.

The result

About £3.65 a contact on 67,600 contacts is roughly £247,000 a month against £52,000 of cost. Scale, with a monthly check on repeat contacts and complaints.

Counting every contact the agent touched, instead of those that stayed resolved, would have overstated the volume by more than 8,000 a month.

From How to Measure AI Value, where the figures are explained in full.

Try it: compute attribution coverage for a portfolio

An insurer has inventoried its AI spend for the year, in £ thousands, and classified each line as outcome-linked (a baseline, an owner and an outcome measure), activity-linked (usage data only) or unattributed.

Figures for the exercise
Direct consumption	300 outcome-linked, 500 activity-linked, 200 unattributed
Licensed seats	100, 900, 400
Embedded tiers	0, 200, 600
Infrastructure	800, 400, 100

Enter the figures, or open the calculator with them loaded.
Read off total AI spend, attribution coverage and unevidenced spend.
Identify which category does most to hold coverage down, and the first action you would take.
Suppose someone reports coverage using only the £3.7m that has been allocated to business units as the denominator. Why is that misleading?

Open the attribution coverage calculator with these figures(opens in a new tab)

Show the worked answer

Total AI spend £4.5m. Outcome-linked £1.2m, so attribution coverage is about 27%. Unevidenced spend is £3.3m.

Licensed seats are the largest unevidenced block (£1.3m), mostly activity-linked: seat usage without outcome measures. Embedded tiers are worst proportionally (no outcome-linked spend at all). A sensible first action is baselines for the two or three largest seat-based workloads, while asking suppliers to disclose the AI share of embedded tiers.

A smaller denominator flatters the result. Coverage must divide by all identified AI spend, and state which test (baseline and outcome measure, or decision-grade evidence) it uses.

Version one of attribution coverage is allowed to be low. It must be honest about its denominator.

Check your understanding

Five practice questions. The final exam uses different questions in the same style.

Progress is saved in this browser only.