Skip to content

Glossary entry

RAG

Retrieval-augmented generation, which supplies a model with documents retrieved at query time so its answers are grounded in the organisation's own information. It avoids retraining, but the retrieved text is billed as input tokens on every call.

Why it matters

RAG is the usual way to make a general model useful on company data. Its cost is easy to underestimate because it sits in input tokens, embeddings and search infrastructure rather than in an obvious line item.

A RAG system searches the organisation's documents for passages relevant to a question and adds them to the prompt. The model then answers from that material, which makes answers more current and easier to check than a model relying on what it learned in training.

The costs sit in four places:

  • Retrieved context. The passages are sent to the model as input tokens on every call. In many deployments they are the largest part of the token bill.
  • Embeddings. Documents are converted into vectors when they are indexed and again when they change.
  • Search infrastructure. A vector database or search service, priced by storage, queries or capacity.
  • Pipelines. Ingesting, cleaning, chunking and refreshing the source documents, which is ongoing engineering work.

The main levers are retrieving fewer, better passages, caching stable context, and measuring whether extra context improves answers enough to justify its cost. More context is not always better: it raises the bill and can lower answer quality.

See Token Economics for how input tokens and caching are billed, and Fine-Tuning for the alternative.

Where this term comes up