The Testing Academy · Masterclass · 3-Part Series
LLM Evaluations for QA, with DeepEval.
Your product now answers in sentences, not status codes. assertEquals can't grade a paragraph, so a new discipline exists: LLM evaluation. This series takes you from "why my Playwright habits don't transfer" to a full pytest-native eval framework grading a live chatbot and a RAG pipeline, the same way you'd grade them in production.
LLM Evaluations for QA
What LLM evaluation is, why classic assertions collapse, what can go wrong in an AI answer, and the one idea that makes it all testable: a second model as the judge.
Concept · what is llm evaluation
Why your assertions stopped working.
Traditional tests check equality. LLM apps need their output scored, not compared.
Everything you've automated so far is deterministic: same input, same output, so assert actual == expected settles it. An LLM breaks that contract on purpose. Ask a chatbot the same question twice and you get two different, both-valid paragraphs. LLM evaluation replaces the equality check with a measurement: a metric reads the answer, scores it between 0 and 1, and compares that score to a threshold you own. The verdict is still pass or fail, so it still belongs in your test suite.
Why it mattersEvery AI feature that ships without evals is tested by your users instead. Evals are the regression suite for the part of the product that writes its own output.
Concept · failure modes
What actually goes wrong in an AI answer.
The bug isn't a stack trace anymore. It's a confident, fluent, wrong sentence.
LLM defects hide in plain sight because the output always looks right. The ones QA hunts for: hallucination (invented facts), irrelevance (fluent answer, wrong question), ungrounded claims (ignores the docs it was given), bias and toxicity (unsafe tone), and PII leakage (repeats data it should mask). None of these throw an exception. All of them are measurable.
Why it mattersYou cannot catch a hallucination with a locator. Naming the failure modes is what turns "the bot feels off" into a test case with a metric attached.
Concept · llm-as-judge
LLM-as-judge: the idea that makes it testable.
A second, trusted model grades the first one, and explains its score in writing.
Human review doesn't scale and string-matching can't read meaning, so evaluation uses a judge LLM: your app answers, the answer is wrapped in a test case, and a metric prompts a different model to score it against a rubric. The judge returns a number and a reason, so a failure reads like a code review comment, not a mystery. The golden rule: the model being graded and the model grading are never the same model. That separation is what makes the verdict independent.
Why it mattersThe score arrives with a written reason. Debugging a failed eval starts from the judge's explanation, which is a luxury a flaky UI test never gave you.
Concept · metric families
The metrics map: floors and ceilings.
Every metric is one of two shapes: a quality you want above a floor, or a risk you want below a ceiling.
Quality metrics score high when the answer is good: answer relevancy (did it answer the question), faithfulness (does it stick to the retrieved context), contextual precision / recall / relevancy (did retrieval fetch the right chunks), and custom G-Eval rubrics like correctness or helpfulness. Safety metrics score high when the answer is bad, so their threshold is a ceiling: hallucination, bias, toxicity, PII leakage. DeepEval flips the comparison for you via is_successful(), you just set the number.
Why it mattersThresholds are your quality contract with the product. "Relevancy ≥ 0.8, hallucination ≤ 0.1" is a testable spec, and tightening it over releases is how AI quality ratchets up.
Concept · the ecosystem
The tools, and why this series bets on DeepEval.
Several good options exist. One of them speaks pytest natively, and that decides it for QA.
The eval ecosystem in one breath: Ragas specialises in RAG metrics, promptfoo does config-driven prompt matrices, LangSmith / Langfuse lean toward tracing and observability, and OpenAI Evals is registry-style benchmarking. DeepEval is the one built like a test framework: metrics are classes, cases are objects, assert_test() is the gate, and everything runs under pytest, so your existing runner, markers, fixtures and CI pipeline work unchanged. Add 14+ research-backed metrics, self-explaining scores, and an optional cloud dashboard (Confident AI) for history and team visibility.
Ragas → RAG-metric specialist, pairs well with notebooks promptfoo → YAML matrices for prompt A/B testing LangSmith → tracing + observability first, evals second OpenAI Evals → benchmark registry, heavier setup DeepEval → pytest-native eval framework: metrics as classes, assert_test() gate, CI-ready, cloud optional Pick DeepEval when your team already lives in pytest and you want AI quality gates in the same pipeline as your other tests.
Why it mattersZero new infrastructure. The eval suite rides the same pytest + CI rails your functional tests already use, so adoption is a pip install, not a platform migration.
DeepEval Setup
Environment, install, keys, your first passing eval test, the two ways to run it, and the graduation step: a real model answering while a different real model judges.
Setup · install
Install DeepEval in a clean room.
One venv, one pinned install, one judge key. Five minutes, done once.
DeepEval is a plain Python package, but two details save you a debugging session: install it in a virtual environment (its dependency tree is heavy), and pin the version your team standardises on so the CLI and metric behaviour stay reproducible. The judge needs an LLM to think with, and out of the box DeepEval judges with OpenAI, so export OPENAI_API_KEY. Keys live in .env, and .env lives in .gitignore, always.
# 1) isolate python3 -m venv venv && source venv/bin/activate # 2) install, pinned (keep the whole team on one version) pip install "deepeval==3.9.9" pytest python-dotenv # 3) the judge thinks with OpenAI by default export OPENAI_API_KEY=sk-... # 4) optional: keys for models you'll put UNDER test later export GROQ_API_KEY=gsk_... # sanity check deepeval --help
Why it mattersAn unpinned eval stack is a flaky eval stack. Judge behaviour shifts across releases; pinning keeps today's green build meaning the same thing next month. (This repo pins 3.9.9 because a later release shipped a broken deepeval test run CLI.)
Setup · first test
Your first eval test: anatomy of a test case.
Four fields and two metrics. If you can write pytest, you already know the shape.
Everything in DeepEval revolves around LLMTestCase: input (what was asked), actual_output (what the app said), expected_output (the reference answer, optional for some metrics), and context (the ground-truth facts the answer must respect, which is what HallucinationMetric grades against). Attach metrics with thresholds, hand it to assert_test(), and it behaves exactly like any other assertion: green or red.
from deepeval import assert_test from deepeval.test_case import LLMTestCase from deepeval.metrics import AnswerRelevancyMetric, HallucinationMetric def test_hello_world(): test = LLMTestCase( input="What is 2+2?", actual_output="4", # what the app answered expected_output="4", # the reference answer # HallucinationMetric grades actual_output against this # grounding context: context=["Basic arithmetic: 2 + 2 = 4."], ) metrics = [ AnswerRelevancyMetric(threshold=0.8), # floor HallucinationMetric(threshold=0.1), # ceiling ] assert_test(test, metrics) # pass/fail, like any assert
Why it mattersThe learning curve is one class and one function. Hard-coding actual_output first isolates the concept; wiring a live model in comes two sections later, and nothing about the test shape changes.
Setup · run & report
Run it two ways, then push the results.
pytest for CI. The deepeval CLI for rich local output and the cloud dashboard.
Because it's pytest underneath, pytest -v just works and is what CI should call. The deepeval test run wrapper adds per-metric scores, reasons and verdict tables in the terminal. When you want history, run deepeval login once with a Confident AI key and every subsequent run is also pushed to a shared cloud dashboard: score trends per metric, pass rates per dataset, and the judge's reasoning for every failure, visible to the whole team instead of one terminal.
# plain pytest, ideal for CI pytest exercises/test_01_basic_answer_relevancy.py -v # deepeval runner: scores, reasons, verdict table deepeval test run exercises/test_01_basic_answer_relevancy.py -d all -v # connect the cloud dashboard (one time) deepeval login # paste your Confident AI API key # same command, results now also land in the dashboard deepeval test run exercises/test_01_basic_answer_relevancy.py
Why it mattersA score nobody can see is a score nobody trusts. Terminal for the loop, CI for the gate, dashboard for the trend: that's the full reporting story in three commands.
Setup · two models, two roles
Real model under test, real judge.
Llama-4 on Groq answers the question. GPT-4.1 grades the answer. Never the same model.
This is the graduation step: replace the hard-coded actual_output with a live call. The model under test (here Groq's Llama-4 Scout, called at temperature=0) produces the answer; the metric's model= parameter picks the judge (GPT-4.1). Two separate keys, two separate roles. Grading a model with itself is like letting a student mark their own exam, so keep the pair split forever.
GROQ_MODEL = "meta-llama/llama-4-scout-17b-16e-instruct" JUDGE_MODEL = "gpt-4.1" def ask_groq(question: str) -> str: client = OpenAI(api_key=os.environ["GROQ_API_KEY"], base_url="https://api.groq.com/openai/v1") resp = client.chat.completions.create( model=GROQ_MODEL, messages=[{"role": "user", "content": question}], temperature=0.0) return resp.choices[0].message.content.strip() def test_groq_llama4_basic_math(): question = "What is 2+2? Reply with just the number." answer = ask_groq(question) # live model answers case = LLMTestCase( input=question, actual_output=answer, expected_output="4", context=["Basic arithmetic fact: 2 + 2 = 4."]) assert_test(case, [ AnswerRelevancyMetric(threshold=0.8, model=JUDGE_MODEL), HallucinationMetric(threshold=0.3, model=JUDGE_MODEL), ]) # GPT-4.1 judges
Why it mattersThis is the exact pattern you'll use on production bots: your app answers over HTTP, an independent judge scores it, and the metric's model= parameter is the only wiring involved.
The Framework & RAG Automation
Scaling from one file to a harness: a metric registry, switchable judges, golden datasets, a live RAG pipeline graded stage by stage, and the whole thing wired into CI.
Framework · architecture
From one test file to an eval framework.
One metric registry, a switchable judge, three live targets. That's the whole architecture.
Scale changes the shape. The framework in this repo grades three apps under test: a React + FastAPI e-commerce chatbot, a full RAG pipeline, and a live third-party bot reached only over HTTP (a true black box, plain-text replies, unknown model, exactly like auditing a vendor's chatbot). One registry of 29 metric rows is the single source of truth, and it drives two surfaces: the pytest suites for CI and an interactive dashboard for demos. The judge is one env var: JUDGE_PROVIDER=openai|groq|ollama, because all three expose OpenAI-compatible endpoints, one CompatibleJudge class serves them all, including a fully offline local judge via Ollama.
# pick the judge with one env var export JUDGE_PROVIDER=openai # or groq, or ollama (fully local) # grade one target's whole suite pytest tests/chatbot/ -v # or slice by marker across the registry pytest -m "chatbot and quality" -v pytest -m "rag and safety" -v # or teach/demo with the interactive dashboard uvicorn dashboard.app:app --port 8203
Why it mattersThe registry pattern is what keeps 29 metrics maintainable: tests and dashboard read the same rows, so adding a metric is one entry, not a hunt through duplicated logic.
Framework · rag evaluation
Evaluating a RAG pipeline, stage by stage.
RAG has two places to fail: retrieval fetches the wrong chunks, or generation ignores the right ones.
A RAG answer is only as good as its weakest stage: ingest → chunk → embed → store → retrieve → answer. So the app under test must expose its retrieval: the RAG Explorer in this repo returns the ranked chunks alongside every reply, on purpose. That unlocks the RAG metric quartet: contextual precision / recall / relevancy grade the retriever (did the right chunks come back, in the right order), while faithfulness grades the generator (does the answer stick to those chunks). A failing metric now points at a stage, not at a vibe.
# the RAG app returns the reply AND the retrieved chunks resp = requests.post("http://localhost:8202/api/chat", json={"message": question, "top_k": 4}).json() case = LLMTestCase( input=question, actual_output=resp["reply"], retrieval_context=resp["retrieval_context"], # ranked chunks ) assert_test(case, [ FaithfulnessMetric(threshold=0.7, model=judge), # generator ContextualRelevancyMetric(threshold=0.7, model=judge), # retriever ])
Why it matters"The RAG bot is wrong" becomes "recall dropped, the retriever misses the refund chunk". Stage-level metrics turn RAG debugging from guesswork into a bisect.
Framework · automation
Goldens, markers, CI: making it run itself.
A golden dataset is your requirement spec, executable. CI runs it on every change.
A golden is one curated example: input, expected behaviour, and the context or sources the answer should draw from. Goldens are data, tests are loops over that data: this repo keeps 19 chatbot goldens (plus 13 adversarial safety probes), 8 RAG goldens with expected sources, and 14 for the live vendor bot, including PII probes. Slice them with pytest markers (chatbot, rag, quality, safety) and wire the slices into CI: cheap smoke slice on every PR, full sweep nightly. From then on a prompt tweak, model upgrade, or retriever change gets graded before users ever see it.
# a golden: one curated, versioned example GOLDENS = [ { "input": "How do I return a damaged item?", "expected_output": "Explain the return window and steps.", "context": ["Returns accepted within 30 days..."], "tags": ["chatbot", "quality"], }, # ...adversarial safety probes live here too ] # .github/workflows/ci.yml, the eval gate - name: LLM eval gate run: pytest -m "chatbot and quality" -v env: JUDGE_PROVIDER: openai OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
Why it mattersThis is the endgame: AI quality as a merge gate. The same discipline you apply to functional regressions now covers hallucination, grounding and safety, automatically, on every change.