Layer A · LLM Observability & Evaluation
Ragas
Open-source evaluation framework with metrics for retrieval-augmented generation, agents and tool use. Its role is quality evidence, not financial management.
Last checked . Visit Ragas
At a glance
- Category
- LLM Observability & Evaluation
- Control layer
- A: Build and run
- Five-Layer model
- Usage Transparency, Output Quality
- Status
- Active
- Typical buyers
- Engineering and platform, AI leadership (CAIO)
- Delivery
- Open source
- Evaluation
Spend controls it documents
Not a spend control
Buyer fit
Include it when the question is whether a cheaper retrieval set-up, model or prompt still meets the quality bar. It produces the quality half of a cost-per-quality comparison; cost has to come from a gateway, observability tool or billing data.
Market outlook
Widely used open-source library. Low lock-in; the main risk is metric drift between versions, so pin versions for comparisons over time.
What to check
- Metrics cover faithfulness, context precision, tool-call accuracy and agent goal accuracy, among others.
- No cost metrics: pair it with a source of cost data.
- LLM-judged metrics add their own model cost to every evaluation run.
Sources
The map describes vendors from public sources; it does not rate, rank or recommend them, and no listing is paid for. Confirm pricing and capabilities with the vendor. See how the map is built.
By Ed Epstein, who works at IBM. The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. About and disclosure · Methodology