DeepEval is an open-source framework that makes LLM evaluation feel like pytest: you write an LLMTestCase, pick a metric like G-Eval or faithfulness, and assert on it. This guide builds a QA eval suite, with a playable diagram and a six-stage roadmap.
Pick a flow and press Play. Each step lights up one piece, from a test case to a pytest assertion, so you can see how an LLM output earns a pass or fail like any unit test.
A DeepEval eval is a test case scored by a metric. You build an LLMTestCase, a metric calls a judge model to score it 0 to 1 against your criteria, and assert_test turns that score into a passing or failing pytest.
You adopt DeepEval one metric at a time. Write one test case, score it, add a rubric, then RAG metrics, datasets, and a CI gate.
DeepEval is an open-source framework for testing the output of LLM applications. If you have ever written a Playwright or JUnit test, its shape will feel familiar on sight: you construct a case, you assert something about it, and a runner tells you pass or fail. The twist is what you are asserting on. A normal test checks a deterministic value, expect(total).toBe(42). An LLM does not return 42; it returns a paragraph, and the same prompt can return a slightly different paragraph tomorrow. DeepEval exists for exactly that gap: it lets you assert on meaning, grounding, and quality instead of on an exact string.
The reason a QA engineer should care is that LLM features are shipping into products faster than anyone is testing them. A support bot, a RAG search box, an AI code reviewer, a test-case generator: each one is a function whose output nobody is regression-testing, because the old tools cannot. Write assert "flaky" in output and it passes on a lucky phrasing and fails the next day when the model says "intermittent" instead. DeepEval replaces that brittleness with a metric: a scored, thresholded judgement of whether the output is relevant, faithful to its sources, correct against a reference, or matching a rubric you wrote in plain English.
Three concrete situations make it land for a test team:
| QA situation | The brittle way | The DeepEval way |
|---|---|---|
| A prompt change ships | Read 30 outputs by hand, hope you notice a regression | Run the same 30 cases through a G-Eval rubric; a dropped score fails CI |
| A RAG answer box | Spot-check that answers "look right" | FaithfulnessMetric checks every answer against its retrieved context |
| An agent that files bugs | Notice weeks later it stopped adding severity | A metric on the structured output catches the drift on the next run |
DeepEval is pip-installable, runs locally, and plugs straight into pytest, so it lives next to the tests you already own rather than in a separate tool nobody opens. It has an optional cloud dashboard (Confident AI) for tracking scores over time, but nothing forces you into it. The mental shift is the whole point: you stop treating LLM output as untestable and start treating it as a value with properties you can assert on. An SDET already has the instinct for that. You know how to turn "it should work" into a specific, checkable claim. DeepEval just gives you claims that survive non-determinism.
DeepEval has a small number of moving parts, and they map cleanly onto ideas you already use. There is the thing under test, the case that describes one interaction with it, the metric that judges the case, and the assertion that turns a judgement into a red or green result. Hold those four in your head and the rest is detail.
The case is an LLMTestCase. It is a plain data object describing one input and what your system did with it:
from deepeval.test_case import LLMTestCase test_case = LLMTestCase( input="Why did the checkout test fail last night?", actual_output="CheckoutTest failed on a NullPointerException at line 42.", expected_output="A NullPointerException in CheckoutTest, around line 42.", retrieval_context=["CheckoutTest.java:42 threw NullPointerException when cart was empty"], )
Only input and actual_output are required. The other fields exist because different metrics need different evidence. A metric that checks correctness needs an expected_output to compare against. A metric that checks whether a RAG answer is grounded needs the retrieval_context your retriever returned. You fill in the fields the metric you chose actually reads, and leave the rest.
The metric is where the judgement happens. DeepEval ships a library of them: AnswerRelevancyMetric, FaithfulnessMetric, the contextual metrics, and the flexible GEval where you write your own criteria. Most of them work by asking a judge LLM to score the case, which is the part that trips people up at first, so say it plainly: the framework calls a model to grade your model. That judge is configurable; it defaults to an OpenAI model but accepts any model, including a local one through a small adapter class. Each metric returns a score between 0 and 1 and a reason string explaining it, and each has a threshold (default 0.5) that decides pass or fail.
Running one metric by hand looks like this, and it is worth doing once so the machinery is not a black box:
from deepeval.metrics import AnswerRelevancyMetric metric = AnswerRelevancyMetric(threshold=0.7) metric.measure(test_case) print(metric.score) # e.g. 0.92 print(metric.reason) # why the judge gave that score
The assertion is the last piece, and it is the one that makes DeepEval feel like a test framework rather than a scoring library. assert_test(test_case, [metric]) runs every metric against the case and raises if any of them fall below threshold, exactly like a failing assert. Wrap that in a pytest function and you have a test that a CI runner understands. For running many cases at once there is evaluate(), and for storing reusable inputs there are datasets and goldens. But the four-part spine, case to metric to score to assertion, is the whole architecture. Everything else is a bigger library of metrics or a nicer way to run them in bulk.
You do not adopt DeepEval by writing a hundred evals on day one. You adopt it one metric at a time, and each stage earns the next. Read these six as gates, not as a menu: a metric you cannot run locally is not ready for a rubric, and a rubric you have not calibrated is not ready for CI.
Stage 1, install and log in. pip install -U deepeval, set your judge model key in the environment, and optionally run deepeval login to connect the Confident AI dashboard. Ship signal: deepeval test run test_smoke.py runs a one-test smoke file green.
Stage 2, your first test case. Build one LLMTestCase from a real interaction and score it with one prebuilt metric, AnswerRelevancyMetric is the gentlest start. Ship signal: you see a score and a reason for an output you actually recognise.
Stage 3, a G-Eval rubric. Write your own criteria in plain English with GEval, because the interesting QA properties, "does the triage note name a root cause", are rarely a prebuilt metric. Ship signal: the rubric passes your good examples and fails a deliberately bad one.
Stage 4, RAG metrics. If your feature retrieves context, add FaithfulnessMetric and the contextual metrics so an answer that contradicts its sources fails even when it reads well. Ship signal: a contradicting answer scores low on faithfulness.
Stage 5, datasets. Move from one case to many with an EvaluationDataset of goldens and a single evaluate() call, so a whole suite runs at once. Ship signal: a table of scores across dozens of cases, not one.
Stage 6, gate CI. Run deepeval test run in your pipeline so a metric below threshold fails the build like any red test. Ship signal: a pull request that degrades quality goes red before it merges.
No stage is large. The discipline is doing them in order, so that by the time an eval blocks a merge, you already trust it.
Getting to a first green eval takes about five minutes. Install the package and set the key for whatever model will act as the judge.
pip install -U deepeval # the judge model reads its key from the environment export OPENAI_API_KEY="sk-..." # optional: connect the Confident AI dashboard to track runs deepeval login
The judge does not have to be an OpenAI model. Any model works through a small adapter, and for cost-sensitive or air-gapped teams a local model via Ollama is a common choice. What matters for now is that some model is reachable, because every LLM-graded metric will call it.
Write the first eval as a real pytest file. Nothing about it is special to DeepEval except the metric and the assertion; the file name, the test_ prefix, and the runner are all ordinary pytest.
# test_triage.py from deepeval import assert_test from deepeval.test_case import LLMTestCase from deepeval.metrics import AnswerRelevancyMetric def test_triage_answer_is_relevant(): case = LLMTestCase( input="Summarise why the checkout suite went red.", actual_output="Three tests failed on a timeout in the payment step; likely a slow staging deploy.", ) assert_test(case, [AnswerRelevancyMetric(threshold=0.7)])
Run it through the DeepEval plugin, which wraps pytest and adds a readable eval report:
deepeval test run test_triage.py
You will see the metric score, the threshold it was measured against, and, when a test fails, the judge's reason for the low score printed right next to it. That last part is the habit-forming detail: a failing eval does not just say "false", it tells you why the output fell short, which is far more actionable than a bare assertion error. Change the actual_output to something evasive and off-topic, rerun, and watch it go red with an explanation. That loop, edit the output, rerun, read the reason, is how you build trust in a metric before you let it gate anything.
Let us build something a real team would keep: an eval for a bug-triage assistant that reads a failing test and returns a short verdict. The property we care about is not just relevance; it is that the verdict is correct against what a human triager would have said. That calls for a reference, so the case carries an expected_output.
# test_triage_correctness.py from deepeval import assert_test from deepeval.test_case import LLMTestCase, SingleTurnParams from deepeval.metrics import GEval correctness = GEval( name="Triage correctness", criteria=( "Determine whether the actual output identifies the same root cause " "as the expected output, and correctly labels it a flake or a real bug." ), evaluation_params=[ SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT, ], threshold=0.7, ) def test_flake_is_triaged_correctly(): case = LLMTestCase( input="LoginTest failed 3 of 20 runs with a timeout, no code change on its path.", actual_output="Flaky: intermittent timeout, no related change. Quarantine, do not file.", expected_output="A flake caused by a timing issue; quarantine rather than file a bug.", ) assert_test(case, [correctness])
Two things are doing the work here. The criteria is a plain-English rubric; you are describing the property in the same words you would use to brief a junior tester. The evaluation_params tell the metric which fields of the case it is allowed to look at, so a correctness check reads the input, the output, and the reference, and nothing else. Under the hood, GEval asks the judge model to reason step by step about whether the criteria is met and returns a score, which is why it handles fuzzy properties that no exact-match assertion could.
Real suites are not one case. Grade a list of them in one call:
from deepeval import evaluate from deepeval.test_case import LLMTestCase evaluate( test_cases=[ LLMTestCase(input="...", actual_output="...", expected_output="..."), # ... more cases, one per representative failure ], metrics=[correctness], )
For inputs you want to store and reuse across runs, EvaluationDataset(goldens=[Golden(...)]) holds the goldens, and you add the generated test cases at eval time. The pattern generalises to any LLM feature you own. Swap the criteria and you have an eval for whether a generated test case has a clear assertion, whether a summary preserves the numbers, whether a code review comment is specific. The case describes the interaction, the metric describes the property, and the assertion makes it a test. Everything after this is choosing better metrics.
The metric is the eval. Pick the wrong one and a bad output passes; pick the right one and the framework earns its place. DeepEval's metrics fall into two camps, and knowing which camp you are in is most of the skill.
G-Eval is the general-purpose workhorse. You give it a name, a criteria string, and the params it may read, and it turns your rubric into a chain-of-thought score. Reach for it whenever the property is specific to your product and no prebuilt metric names it: "does the triage note propose a next action", "does the release summary mention every breaking change". It is flexible precisely because you write the standard.
The RAG metrics are prebuilt because retrieval-augmented answers fail in a few well-known ways, and each metric names one of them:
| Metric | What it catches | Reads |
|---|---|---|
AnswerRelevancyMetric | An answer that wanders off the question | input, actual_output |
FaithfulnessMetric | An answer that contradicts its retrieved sources | actual_output, retrieval_context |
ContextualPrecisionMetric | Relevant chunks ranked below noise | input, retrieval_context, expected |
ContextualRecallMetric | The retriever missing chunks it needed | expected, retrieval_context |
ContextualRelevancyMetric | Retrieved context that is off-topic | input, retrieval_context |
Faithfulness is the one QA teams reach for first, because a confident, well-written, wrong answer is the failure mode that erodes trust in a RAG box. It scores the answer's claims against the retrieval_context. By default only a claim that contradicts the sources counts against it; pass penalize_ambiguous_claims=True so unsupported claims fail too, and an answer that reads beautifully but is not backed by the chunks goes red:
from deepeval.metrics import FaithfulnessMetric case = LLMTestCase( input="What timeout does the checkout suite use?", actual_output="The checkout suite uses a 30 second timeout.", retrieval_context=["checkout.config: requestTimeout = 10s"], ) FaithfulnessMetric(threshold=0.8, penalize_ambiguous_claims=True).measure(case) # low score: answer says 30s, source says 10s
There is a third, quieter camp worth knowing: deterministic metrics. ToolCorrectnessMetric checks whether an agent called the tools it should have by comparing tools_called with expected_tools, with no judge LLM involved by default, so it never flakes and costs nothing (passing available_tools adds an LLM-judged optimality check). And DAGMetric structures a judgement as a decision tree: the judge model decides at each node, but you map each verdict path to a fixed score, so the scoring itself is deterministic and explainable. The rule of thumb: if a property can be checked by a rule, use a deterministic metric and save the judge for the genuinely fuzzy calls. Every LLM-graded metric is a non-deterministic, metered call; spend them where they are actually needed.
Because a DeepEval eval is a pytest test, wiring it into CI is mostly wiring pytest into CI, with one command swapped. The eval file looks like any test file, and deepeval test run replaces pytest as the invocation so the DeepEval reporter and plugin are active.
# .github/workflows/evals.yml name: llm-evals on: [pull_request] permissions: contents: read jobs: evals: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install -U deepeval - run: deepeval test run tests/evals -n 4 env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
The -n 4 flag runs cases in parallel, which matters because judge calls have latency; four at a time turns a slow suite into a tolerable one. A metric below threshold makes the run exit non-zero, so the pull request goes red exactly like a failing unit test. That is the whole point of living inside pytest: no new gate to teach the team, no separate dashboard to check before merging.
Two operational choices make this durable. First, decide what runs per pull request versus on a schedule. A small, fast, cheap smoke set of evals belongs on every PR; a large, expensive suite over hundreds of goldens belongs on a nightly cron, where its cost and latency do not block a merge. Second, track scores over time. Running deepeval login once sends results to the Confident AI dashboard, so instead of a binary pass or fail you get a trend line, and a slow three-week slide in faithfulness becomes visible before it becomes a customer complaint.
Evals have their own failure modes, and they are not the ones you are used to. The whole value of DeepEval comes from an LLM judging an LLM, and that judge brings non-determinism, cost, and fallibility into your test suite. Manage all three deliberately or the suite becomes noise.
Flaky eval scores. A judge model can score the same output 0.82 one run and 0.78 the next. If your threshold sits at 0.80, that is a flaky test with a new cause. Stabilise it: set thresholds with headroom rather than on a knife-edge, pin the judge model and version so a silent model update does not shift every score, enable caching so identical cases replay instead of re-grading, and lean on G-Eval's chain-of-thought and clear criteria, which produce steadier scores than a vague rubric. Where a property can be checked deterministically, do that instead and remove the judge from the loop entirely.
Do not over-trust the judge. A metric is an opinion from a model, not ground truth. Before an eval is allowed to block merges, calibrate it: label a sample of cases by hand, run the metric over them, and check that it agrees with you. If it passes outputs you consider bad, tighten the criteria or raise the threshold. An uncalibrated eval that everyone trusts is worse than no eval, because it launders a bad output as approved.
Keep test data inside your boundary. Eval fixtures are often real: production transcripts, customer records in a RAG context, internal stack traces. Every LLM-graded metric sends that data to the judge model. If the judge is a hosted API, that data leaves your perimeter. For sensitive fixtures, run the judge on a local model, or scrub the data first. And treat a genuinely flaky eval the way you treat a flaky test: quarantine it, open a ticket, and fix the cause. Never let a team learn to ignore red, because the day they ignore a real regression is the day the eval was pointless.
No, it sits inside them. Your deterministic tests still assert deterministic things. DeepEval adds a way to assert on non-deterministic LLM output, and it runs through the same pytest command, so it extends your suite rather than replacing it.
DeepEval is Python and pytest-native, so it fits a codebase that already tests in Python. promptfoo is config-first, driven by a YAML file, and strong at comparing prompts and providers side by side. OpenEvals is a lighter LLM-as-judge kit from LangChain. They overlap; the deciding factor is usually whether you want your evals as pytest code (DeepEval), as declarative config (promptfoo), or as small judge functions (OpenEvals).
Pin and version the judge model, set thresholds with headroom instead of on the edge, enable caching, write clear G-Eval criteria, and use deterministic metrics where a rule can express the check. Those five together turn a jittery score into a stable gate.
No. The framework is open-source and runs entirely locally. The Confident AI dashboard is an optional hosted add-on for tracking scores over time; you can run every metric and gate CI without ever logging in.
Yes. Point the judge at a local model through the custom model adapter (Ollama is common) and skip deepeval login. Nothing then leaves your machine, which matters when fixtures contain real product data.
Start with five to ten real cases where you already know the right verdict, and one metric. A small, trustworthy set that gates CI beats a large set you have never calibrated. Grow it as you find failures worth pinning.
The six stages, in order, are the map to pin above your desk:
| Stage | What you build | Ship signal |
|---|---|---|
| 1. Install and log in | pip install -U deepeval, judge key in env | a one-test smoke file runs green |
| 2. First test case | one LLMTestCase + AnswerRelevancyMetric | a real score and reason you recognise |
| 3. G-Eval rubric | your own criteria in plain English | it passes good cases, fails a bad one |
| 4. RAG metrics | FaithfulnessMetric and the contextual metrics | a contradicting answer scores low |
| 5. Datasets | an EvaluationDataset + one evaluate() | a table of scores across many cases |
| 6. deepeval test run | the suite in CI on pull requests | a quality regression goes red before merge |
Read the order as a dependency chain. Stage 2 proves a metric can score at all. Stage 3 makes it say something specific to your product. Stage 4 grounds it against sources so confident nonsense fails. Only at stage 5 do you scale to a real suite, and only at stage 6 do you let it block a merge, because by then you trust it. Rushing to stage 6 with an uncalibrated metric is how a team learns to ignore red.
Series siblings: promptfoo for QA and OpenEvals for QA. See also LangSmith for QA, RAG for QA, and the AI for QA hub.