Executive summary
A customer-operations team used AI to write the summary agents normally type after each contact. On simple contacts it helps. On complex ones, such as billing disputes and complaints, agents spend longer correcting the summary than they would have spent writing it. Fully loaded cost is £180k a year. Claimed value is £240k, but only £60k has been booked. Evidence level is E2. Decision: redesign. Narrow the scope, change the summary format, and decide again with fresh numbers.
Situation and intended outcome
About 600,000 contacts a year each end with after-call work. The intended outcome was less time on that work, released as reduced overtime and agency hours.
Economic baseline
| Measure | Before |
|---|---|
| Contacts a year | 600,000 |
| After-call work | The time to write each summary, measured by the telephony system |
| Loaded agent cost | £24 an hour |
AI investment and full cost
| Cost line | Annual (£k) |
|---|---|
| Licences and platform | 70 |
| Model consumption | 35 |
| Integration and engineering (amortised) | 30 |
| Supervisor quality checks | 35 |
| Governance and assurance | 5 |
| Change and training | 5 |
| Fully loaded cost | 180 |
Agent time spent correcting summaries is not in this table. It is netted from the time saving below, so that it is counted once.
Consumption and workflow
The assistant drafts a free-text summary at the end of each contact. The agent reads it, corrects it and saves it. Supervisors sample summaries for accuracy.
Outcome and quality floor
The quality floor was 95% factual accuracy in sampled summaries. Overall accuracy is 91%. The split matters: 97% on simple contact types, 82% on complex ones. On simple contacts the net time saving is clear. On complex ones, correction time exceeds the time the draft saves.
Evidence assessment
A matched-cohort comparison across teams, which fits E2: enough for a portfolio decision, not for a material expansion. The comparison measured net time after correction, which is why the claim is already net of that cost.
Net economic position
| Line | Calculation | Annual (£k) |
|---|---|---|
| Net time saved | 600,000 contacts × 60 seconds = 10,000 hours at £24 | 240 |
| Claimed value | 240 | |
| Realised value | 2,500 hours converted to lower agency and overtime hours, at £24 | 60 |
| Fully loaded cost | 180 | |
| Claimed net | 240 − 180 | +60 |
| Realised net | 60 − 180 | −120 |
Only a quarter of the saved time has become money. The rest is spread thinly across many agents in minutes per shift, and nobody has changed a roster to capture it.
Decision: redesign
The outcome is valuable and the evidence is respectable, but the workflow is weakening the case. More funding would scale a problem. Stopping would discard a clear gain on simple contacts.
| Field | Entry |
|---|---|
| Owner | Head of Customer Operations |
| Evidence level | E2: matched cohort |
| Claimed / realised / cost (£k a year) | 240 / 60 / 180 |
| Decision | Redesign |
| Changes | Use the assistant only for simple contact types. Replace free-text summaries with structured fields to cut correction time. Add a confidence flag that sends low-confidence drafts to the agent unedited |
| Capturing the value | Agree with workforce planning how saved minutes become fewer agency hours, before the next trial |
| Stop rule | If the redesigned version is not net positive on realised value after two quarters, stop |
| Review | 31 January 2027 |
Lessons, limitations and next review
- The aggregate figure hid the real result: a good outcome on one contact type and a poor one on another. Segment before deciding.
- The gap between claimed and realised value was a roster and planning problem, not a model problem.
- Quality checks are a cost line. Correction time belongs in the saving. Keeping them apart avoids double counting.
- The redesigned version needs its own baseline. Do not carry the old numbers forward.
Status: Illustrative decision case, version 1.0, 8 October 2026.