promptfoo is an open-source, config-first tool for testing LLM prompts: you declare prompts, providers, and assertions in one YAML file, run the matrix, and grade every output. This guide builds a QA eval, with a playable diagram and a six-stage roadmap.
Pick a flow and press Play. Each step lights up one piece, from a config file to a graded assertion, so you can see how one YAML file becomes a full LLM test run.
A promptfoo eval is one config file. You declare prompts, providers, and tests; promptfoo runs every combination and grades each output with assertions, from a simple contains to a model-graded llm-rubric.
You adopt promptfoo one assertion at a time. Run one config, add deterministic checks, then model-graded rubrics, provider comparison, and CI.
promptfoo is an open-source, config-first tool for testing LLM prompts. You describe the whole eval in one YAML file, the prompts you want to try, the models to run them against, and the assertions each output must pass, then one command runs the entire matrix and grades every result. If DeepEval feels like pytest for LLM output, promptfoo feels like a parameterised test table: declarative, comparative, and readable by anyone on the team, not just the person who wrote the Python.
The reason a QA engineer should care is comparison. LLM work is full of "which is better" questions that are miserable to answer by hand. Does the new prompt beat the old one? Is the cheaper model good enough to swap in? Did upgrading the provider quietly regress our triage quality? promptfoo answers those by running every prompt against every provider over the same test cases and putting the graded results side by side in one table. That is a test matrix, the oldest idea in QA, applied to prompts and models.
Three situations where it earns its place on a test team:
| QA situation | The manual way | The promptfoo way |
|---|---|---|
| Tweaking a prompt | Eyeball a few outputs, hope the change helped | Run old vs new over 20 cases; the table shows which regressed |
| Choosing a model | Vibes about gpt vs claude vs a local model | Same cases, three providers, graded assertions, real numbers |
| Guarding a release | Nobody checks the bot before deploy | promptfoo eval --fail-on-error in CI fails the merge on a dropped assertion |
It installs with a single command, needs no account, and the config lives in your repo next to the code it tests, so a prompt change and its eval land in the same pull request. The mental model is the smallest possible: one file declares prompts times providers times tests, and the tool grades the grid. An SDET already thinks in matrices and expected results. promptfoo just points that instinct at prompts.
Everything in promptfoo is a promptfooconfig.yaml. There are four keys you will use constantly, and they map onto a test table you have built a hundred times: the thing you are varying (prompts), the environment you run it in (providers), the cases (tests), and the checks (assertions). Learn the shape of the file and you have learned the tool.
prompts: - "Summarise why this CI run failed: {{log}}" providers: - openai:gpt-5-mini - anthropic:messages:claude-sonnet-4-5 tests: - vars: log: "NullPointerException at CheckoutTest.java:42, cart empty" assert: - type: contains value: "NullPointer" - type: llm-rubric value: "Names the empty cart as the likely cause"
The prompts are templates. The {{log}} placeholder is filled from each test's vars, so one prompt covers many inputs. Prompts can be inline strings, or pulled from files with file://prompt.txt, file://prompt.json for chat-format messages, or even file://gen.py:build to generate them in code.
The providers are the models, written as short identifiers: openai:gpt-5-mini, anthropic:messages:..., ollama:chat:llama3.1 for a local model, and many more. List several and promptfoo runs the full cross-product: every prompt, against every provider, over every test. That cross-product is the whole reason the tool exists; it turns "compare these" into a single run.
The tests are the cases. Each has vars that fill the prompt and an assert block that grades the output. There is also a defaultTest key for assertions you want applied to every case, so a rule like "output must be valid JSON" is written once, not per test.
The assertions are where pass and fail are decided, and they come in two families that matter enormously in practice: deterministic checks that are fast, free, and never flake (contains, equals, is-json, regex, custom javascript), and model-graded checks that ask a grader LLM to judge fuzzy properties (llm-rubric, g-eval, factuality). You run it all with three commands: npx promptfoo@latest init to scaffold, promptfoo eval to run the grid, and promptfoo view to open the results in a browser. No test class, no runner to learn, just a file and a command.
You adopt promptfoo one assertion at a time, not by writing an exhaustive config on day one. Read these six stages as gates: a config you cannot run is not ready for asserts, and an eval with no model-graded check is not yet testing the things that actually matter.
Stage 1, install and init. npx promptfoo@latest init scaffolds a config and a README in seconds. Ship signal: promptfoo eval runs the starter config and promptfoo view shows a table.
Stage 2, your first real config. Replace the sample with one prompt, one provider, and one test drawn from a real feature you own. Ship signal: you see your own output graded, not the sample's.
Stage 3, deterministic asserts. Add the free, fast checks first, contains, equals, is-json, a small javascript function. Ship signal: a malformed output fails without any model call.
Stage 4, model-graded asserts. Add llm-rubric or g-eval for the fuzzy properties a rule cannot express. Ship signal: a plausible-but-wrong answer fails on the rubric.
Stage 5, compare providers. Add a second and third provider and read the side-by-side table in promptfoo view. Ship signal: a real, evidence-backed answer to "which model should we use".
Stage 6, CI and red team. Run promptfoo eval in the pipeline to gate a merge, and try npx promptfoo@latest redteam init then npx promptfoo@latest redteam run for adversarial cases. Ship signal: a prompt regression goes red before it ships.
Each stage is small, and the order matters: deterministic before model-graded, one provider before three, local run before CI.
promptfoo needs Node 20.20 or 22.22 and newer (24 LTS is recommended) and nothing else to start. The init command scaffolds a working config so you are running an eval within a minute.
# scaffold a config in the current directory npx promptfoo@latest init # provider keys come from the environment export OPENAI_API_KEY="sk-..."
Open the generated promptfooconfig.yaml and replace it with a real QA case. Here is a triage eval: given a failing-test log, the model should summarise the failure, and we assert on both a hard fact and a fuzzy judgement.
# promptfooconfig.yaml prompts: - "You are a QA triage bot. Summarise this failure in one line: {{log}}" providers: - openai:gpt-5-mini tests: - vars: log: "PaymentTest timed out after 30s on staging, 3 of 3 runs" assert: - type: icontains value: "timeout" - type: llm-rubric value: "Identifies a timeout, and suggests staging or environment as a likely cause"
Run the grid, then open the report:
npx promptfoo@latest eval npx promptfoo@latest view
The terminal prints a pass or fail per assertion; view opens a web table with the output, each assertion's result, and the grader's reasoning for the model-graded one. Change the prompt to something vague, rerun, and watch the rubric fail with an explanation. That edit-run-read loop, in the browser table, is how you feel out whether a rubric is strict enough before you trust it. By default promptfoo caches results, so an unchanged case is not re-run; pass --no-cache when you want a clean pass.
{{vars}} and put the variety in the tests list. Ten cases against one prompt is a real regression set; ten copies of a prompt is a maintenance headache.Let us grow the starter into something a team keeps: a triage eval that runs many real failures, shares a couple of assertions across all of them, and compares two models. This is where the config file earns its keep, because adding a case is one entry and adding a model is one line.
# promptfooconfig.yaml description: "CI failure triage eval" prompts: - file://prompts/triage.txt providers: - openai:gpt-5-mini - anthropic:messages:claude-sonnet-4-5 # applied to every test below defaultTest: assert: - type: llm-rubric value: "Classifies the failure as either a flake or a real bug, and says which" tests: - vars: log: "LoginTest failed 3 of 20 runs, timeout, no code change on its path" assert: - type: icontains value: "flak" - vars: log: "CheckoutTest NPE at line 42 on every run since the cart refactor" assert: - type: icontains value: "bug"
Two design moves are doing the work. The defaultTest block carries the assertion that applies to every case, the flake-or-bug classification, so you write it once. Each test then adds only its own specific check. And listing two providers means every case runs against both models, so the report is a direct comparison over identical inputs. Pull the prompt itself into prompts/triage.txt so it is versioned and reviewable on its own.
Run it and read the table by provider. If one model consistently fails the rubric on real bugs, you have your answer about which to ship, backed by graded evidence rather than a hunch. Add cases as you find failures worth pinning, exactly as you would grow any regression suite. The config stays flat and readable no matter how many rows it holds, which is the property that makes teams actually maintain it.
The assertion is where promptfoo decides pass or fail, and choosing well is most of the skill. There are two families, and the guiding rule is simple: use a deterministic check whenever a rule can express the requirement, and reach for a model-graded check only for genuinely fuzzy properties.
Deterministic assertions are fast, free, and never flake, because no model is involved. They cover a surprising amount:
assert:
- type: contains
value: "severity"
- type: is-json # the output must parse as JSON
- type: regex
value: "P[0-4]" # a priority label is present
- type: javascript
value: "output.length < 280" # fits a Slack message
- type: latency
threshold: 3000 # under 3s (run eval with --no-cache so latency is real)
The javascript assertion is the escape hatch: your function receives the output and returns a boolean, a 0-to-1 score, or a full grading result, so any check you can write in code is available. Prefer these; a suite built on deterministic asserts runs in milliseconds and costs nothing.
Model-graded assertions ask a grader LLM to judge what a rule cannot:
| Assertion | What it checks |
|---|---|
llm-rubric | Output meets a plain-English rubric you write |
g-eval | Chain-of-thought scoring against criteria |
factuality | Output agrees with a reference answer |
context-faithfulness | A RAG answer is supported by its context |
answer-relevance | Output actually addresses the question |
similar | Embedding cosine similarity above a threshold (an embedding model, not an LLM judge) |
assert:
- type: llm-rubric
value: "Proposes a concrete next action, not just a description"
threshold: 0.8
- type: similar
value: "quarantine the flaky test"
threshold: 0.75
Each takes a value, an optional threshold, and an optional provider to pin which model grades. The trade is real: model-graded asserts test the things you actually care about, but they cost tokens and can wobble run to run. Build the deterministic wall first, then add model-graded checks only where intent cannot be captured by a rule.
Because a promptfoo eval is one command over one config file, CI integration is short. Run promptfoo eval --fail-on-error in the pipeline; with that flag a failed assertion makes the run exit non-zero, which fails the job and blocks the merge like any red test. Without it, promptfoo reports the failures but still exits 0, so the flag is not optional for a gate.
# .github/workflows/prompt-evals.yml name: prompt-evals on: [pull_request] permissions: contents: read jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: "24" - run: npx promptfoo@latest eval -c promptfooconfig.yaml --fail-on-error --output results.json env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
The --output results.json writes the full result for archiving or a later diff. There is also an official GitHub Action that comments the eval table on the pull request, so reviewers see which cases moved without leaving the diff. As with any LLM eval, decide what runs per pull request, a small fast set, versus on a nightly schedule, the big expensive grid, so judge cost never blocks a merge.
promptfoo also does red teaming, which is adversarial QA for LLM apps. npx promptfoo@latest redteam init and npx promptfoo@latest redteam run generate attack cases, jailbreak attempts, prompt injection, PII extraction, harmful-content coaxing, and run them against your app so you find the holes before an attacker does. For a QA team shipping a customer-facing bot, that is the security test pass you did not know you could automate.
promptfoo removes a lot of manual toil, but its model-graded side brings the same three hazards every LLM eval has: flakiness, cost, and a judge that can be wrong. Handle them on purpose.
Flaky assertions. A model-graded check can pass at 0.82 and fail at 0.78 on the same output. Reduce the wobble: pin the grader model and version, set thresholds with headroom instead of on the edge, keep caching on so unchanged cases are not re-graded, and write tight, specific rubrics, because a vague rubric grades inconsistently. Lean on deterministic asserts wherever possible; they never flake at all, which is the strongest guardrail there is.
Do not over-trust the grader. An llm-rubric result is one model's opinion, not truth. Before it gates merges, calibrate: run it over a handful of outputs you have judged yourself and confirm it agrees. If it passes answers you consider wrong, tighten the rubric or raise the threshold. A confident, wrong grader that everyone trusts is worse than no gate.
Keep test data inside your boundary. promptfoo sends your prompts, your test vars, and the outputs to whichever providers and grader you configured. If those are hosted APIs, real fixtures, customer transcripts, internal logs, leave your perimeter. For sensitive data, run against a local model through the ollama provider and grade with a local grader, or scrub the fixtures first. And treat a genuinely flaky case like a flaky test: quarantine it, file it, fix the cause, and never train the team to wave past a red eval.
No. Your deterministic unit and integration tests are unchanged. promptfoo adds a layer that tests LLM output, prompts, and model choices, in a config file that runs alongside them. It complements the suite rather than replacing any of it.
promptfoo is config-first: the eval is a YAML file, and its superpower is comparing prompts and providers side by side in a matrix. DeepEval is pytest-native Python code. OpenEvals is a lighter LLM-as-judge kit from LangChain. Pick promptfoo when you want declarative config and provider comparison; pick the others when you want your evals as code.
Prefer deterministic assertions, and for model-graded ones pin the grader model, set thresholds with headroom, keep caching on, and write specific rubrics. Deterministic-first is the single biggest lever.
No. promptfoo is open-source and runs locally; you only pay for the model provider calls the eval makes. There is a hosted option for teams, but the CLI, the config, CI integration, and the web view all work without an account.
Yes. Use the ollama provider for both the model under test and the grader, and nothing leaves your machine. That is the right setup when test fixtures contain real product data.
Start with five to ten real cases and one prompt, mostly deterministic asserts plus one rubric. A small, calibrated config that gates CI beats a giant grid you have never checked. Grow it from real failures.
The six stages, in order, are the map to keep by the config:
| Stage | What you build | Ship signal |
|---|---|---|
| 1. Install and init | npx promptfoo@latest init | eval runs, view shows a table |
| 2. First config | one prompt, one provider, one real test | your own output graded |
| 3. Deterministic asserts | contains, is-json, javascript | a malformed output fails, no model call |
| 4. Model-graded | llm-rubric, g-eval | a plausible-but-wrong answer fails |
| 5. Compare providers | two or three providers + promptfoo view | an evidence-backed model choice |
| 6. CI and red team | promptfoo eval --fail-on-error in CI, promptfoo redteam run | a prompt regression goes red before merge |
Read the order as a dependency chain. Stage 3 builds a wall of free, deterministic checks. Stage 4 adds judgement only where a rule cannot reach. Stage 5 turns the matrix into a real model decision. Only at stage 6 do you let it block a merge, because by then the config is calibrated and cheap to run. Rushing to CI with one untested rubric is how a team ends up ignoring a red eval.
Series siblings: DeepEval for QA and OpenEvals for QA. See also LangSmith for QA, LangChain for QA, and the AI for QA hub.