Is Promptfoo open source?

Yes, Promptfoo is open source under the MIT License and is actively maintained with community contributions.

Does Promptfoo share my API keys or data?

By default, Promptfoo runs locally. Your API keys are read from environment variables and calls are made directly from your machine to the LLM provider.

Which LLM providers are supported?

Promptfoo supports over 60 providers, including OpenAI, Anthropic, Azure, Google Gemini, Hugging Face, and local providers like Ollama.

What is LLM Red Teaming in Promptfoo?

It is an automated security scanning feature that generates adversarial probes to test your AI for vulnerabilities like prompt injection, harmful content, and data leakage.

How do I integrate Promptfoo into my workflow?

You can use it as a CLI tool, a Node.js library, or integrate it into your CI/CD pipeline using the official GitHub Action to run evals on every PR.

Can Promptfoo test RAG pipelines?

Yes, Promptfoo includes specific tools and guides for evaluating Retrieval-Augmented Generation (RAG) performance and retrieval accuracy.

Nobody owns this listing yet

Promptfoo is live in the directory, but no verified owner controls the page.

Claim this listing

Pick your launch day — Schedule a launch and compete for the daily #1 spot.
Win trophies you can embed — Rank top 3 on launch day and get an embeddable trophy.
“Featured on AI Kaptan” badge — Owner-only badge with your live upvote count.
Control what the page says — Edit copy, pricing, links and screenshots as the owner.

Unclaimed listings don't stay up forever. Listings left unclaimed are removed from the directory in periodic cleanups. When a listing goes, it goes everywhere — the tool page and every alternatives comparison it appears on, along with the rankings and backlinks that page had built up.

Verify with a work email on the domain (about 2 minutes), then a one-time $5 listing fee. See everything you get

Developer Tools

Freemium

Promptfoo

Open-source CLI and library for evaluating LLM prompt quality, performance, and security. Compare models side-by-side, automate assertions, and run adversarial red teaming in CI/CD.

Visit WebsiteView Alternatives

—

Rating

Reviews

Upvotes

3 months ago

Listed

Prompt Engineering

LLM Testing

AI Security

Red Teaming

RAG Evaluation

Model Benchmarking

DevOps

Open Source

Tool information

Provider

Promptfoo (OpenAI)

Platforms

Web

Linux

macOS

Windows

Languages

English

API Available

Yes

Added to directory

3 months ago

Last updated

about 1 month ago

Rating

Not available

Pricing

Freemium

About Promptfoo

What the tool does and who it's for

Promptfoo is a professional-grade, open-source testing framework designed to evaluate and secure LLM applications. It allows developers to systematically test prompts, models, and RAG pipelines against a suite of test cases. By using a declarative YAML configuration, users can run side-by-side comparisons of multiple models (including OpenAI, Anthropic, Gemini, and Llama), define automated assertions for quality and cost, and integrate security red teaming to identify vulnerabilities like jailbreaks or PII leaks. Since March 2026, Promptfoo has been part of OpenAI while remaining MIT-licensed and provider-agnostic.

Key capabilities

Side-by-side matrix view for comparing outputs from multiple LLM providers simultaneously.

Automated assertions including semantic similarity, regex, JSON schema, and LLM-graded rubrics.

Integrated Red Teaming module with 100+ attack plugins to scan for jailbreaks and PII leaks.

Support for 60+ providers including OpenAI, Anthropic, Google Gemini, and local models via Ollama.

Local-first architecture ensuring API keys and evaluation data remain on your machine by default.

CI/CD integration via official GitHub Actions to test prompts before every pull request merge.

Synthetic dataset generation to expand test cases based on existing prompts and failure modes.

Caching and concurrency features to speed up large-scale evaluation runs and reduce API costs.

Pricing

Freemium — plans below

Community

Free

Open-source (MIT License)
All LLM evaluation features
All model providers
10k red teaming probes/month
100% local execution
Community support

Team

Enterprise

Contact sales

Custom red teaming limits
SSO & granular permissions
Continuous monitoring & alerts
On-premise deployment options
Dedicated deployment engineer

Reviews

Be the first to review Promptfoo

No reviews yet

Be the first to share your experience with Promptfoo.

Featured Tools

Handpicked by our team of experts

Suggest an Update

Found outdated information? Help us keep this listing accurate.

FAQ

Promptfoo FAQs

Common questions about Promptfoo