Layer A · LLM Observability & Evaluation
Promptfoo
Part of OpenAI
Open-source test harness that compares prompts and models on output quality, estimated cost and latency, with red teaming for security.
Last checked . Visit Promptfoo
At a glance
- Category
- LLM Observability & Evaluation
- Control layer
- A: Build and run
- Five-Layer model
- Usage Transparency, Output Quality
- Status
- Acquired by OpenAI
- Typical buyers
- Engineering and platform, AI leadership (CAIO)
- Delivery
- Open source
- Evaluation
- Red teaming
Spend controls it documents
Not a spend control
Buyer fit
Useful before a model or prompt change ships: run the same test cases across candidates and see what quality each one buys at what cost. Cost assertions fail a test when a response is too expensive; they do not cap spending in production. Best owned by engineering, with results shared with whoever approves model choices.
Market outlook
OpenAI announced on 9 March 2026 that it will acquire Promptfoo and fold it into its Frontier agent platform, while keeping the open-source project and support for other providers. Watch whether multi-provider testing stays first-class.
What to check
- Cost and latency assertions apply to completed responses. They are a test gate, not a spend control.
- Configuration is plain files in the repository, so switching cost is low.
- Check how the pending OpenAI ownership affects support for competing model providers.
Ownership and name changes
9 March 2026
OpenAI agrees to acquire Promptfoo
The open-source evaluation and red-teaming tool is to be folded into OpenAI Frontier; OpenAI says the open-source project will continue. OpenAI announcement
Sources
- Promptfoo: getting started
- OpenAI to acquire Promptfoo, 9 March 2026
- Promptfoo press page: "Promptfoo is now part of OpenAI" (checked 28 Oct 2026; no closing date given)
The map describes vendors from public sources; it does not rate, rank or recommend them, and no listing is paid for. Confirm pricing and capabilities with the vendor. See how the map is built.
By Ed Epstein, who works at IBM. The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. About and disclosure · Methodology