DeepEval: a threshold where assertEquals was

01From assertEquals to assert_test

Arrange and act stay ordinary pytest. Only the assertion changes: assert_test(case, [metric]) sits where assert actual == expected used to, so CI needs no new runner. What is new is the test case itself. The chapter's slide maps each DeepEval parameter to the QA idea it replaces.

02The two tests

The chapter folder holds two test files, setup notes and two slides.

test_01 is the whole idea in one function. Nothing calls a model under test: the answer is typed in, and only the judge is called.

# Setup: DeepEval needs a "judge" LLM to score the output. Pick one.
# Groq (cheap, OpenAI-compatible endpoint -> registered as a local model):
#   deepeval set-local-model --model openai/gpt-oss-120b \
#       --base-url "https://api.groq.com/openai/v1" --format json --prompt-api-key
# OpenAI:
#   export OPENAI_API_KEY=your_key_here
#   deepeval set-openai --model gpt-4o-mini
# Note: `deepeval set-grok` is xAI's Grok, NOT Groq.com.
# Run: pytest test_01_Anwser_Relevancy.py


from deepeval.test_case import LLMTestCase
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric


def test_hello_world():

    test = LLMTestCase(
        input="What is 2+2?",
        actual_output="4",
        expected_output="4",
        context=["Basic arithmetice perform and give result"]
    )
    metric = [     AnswerRelevancyMetric(threshold=0.9)]
    assert_test(test, metric)

test_02 adds a real subject model. Note where the call happens: at module level, so pytest asks Groq while it is still collecting the file.

question = "What is 2+2? Reply with just the number."
answer = llm_response(question)
print(f"\n[Groq {GROQ_MODEL}] → {answer!r}\n")



def test_l4_with_judge_qwen():
    case = LLMTestCase(
        input=question,
        actual_output=answer,
        expected_output="4",
        # HallucinationMetric scores actual_output against this grounding text.
        context=["Basic arithmetic fact: 2 + 2 = 4"],
    )

    metrics = [
        AnswerRelevancyMetric(threshold=0.8, model=JUDGE_MODEL),
        HallucinationMetric(threshold=0.3, model=JUDGE_MODEL),
    ]
    assert_test(case, metrics)

sequenceDiagram
    participant P as pytest
    participant S as Subject qwen3.8-27b on Groq
    participant D as DeepEval
    participant J as Judge gpt-oss-120b on Groq
    P->>S: at import, ask the question at temperature 0
    S-->>P: raw answer text
    P->>D: assert_test with the LLMTestCase and two metrics
    D->>J: Answer Relevancy prompts
    J-->>D: score and reason
    D->>J: Hallucination prompts, against the context
    J-->>D: score and reason
    D-->>P: pass only if every score meets its threshold

03Live demo: build a test case and run it

Pick a case, choose which fields go into the LLMTestCase, tick metrics and set thresholds, then press Run pytest. The pass and fail rules are real: a metric errors when a field it reads is missing, and a score passes when it is at or above its threshold. The scores are fixed per case. The repo records no scores for this chapter, so they are illustrative, except Faithfulness and Correctness on the refund case, which chapter 16 recorded for that exact reply.

04Wire the judge

The judge is configured once per machine. DeepEval defaults to OpenAI. Groq speaks the OpenAI API, so the course registers it as a "local model" and grades with openai/gpt-oss-120b for a fraction of the cost.

flowchart TD
    T["pytest test_01_Anwser_Relevancy.py"] --> TC["LLMTestCase<br/>input + actual_output + context"]
    TC --> M["AnswerRelevancyMetric&#40;threshold=0.9&#41;"]
    M --> R{"Which judge?<br/>.deepeval provider flag"}
    R -->|USE_OPENAI_MODEL| O["api.openai.com<br/>gpt-4o-mini"]
    R -->|USE_LOCAL_MODEL| G["api.groq.com/openai/v1<br/>openai/gpt-oss-120b"]
    R -->|USE_OLLAMA_MODEL| L["localhost:11434<br/>free, slower"]
    O --> SC["Score 0.0 - 1.0"]
    G --> SC
    L --> SC
    SC -->|">= 0.9"| PASS["Test passes"]
    SC -->|"< 0.9"| FAIL["Test fails<br/>+ the judge's reason"]

### Pick the judge LLM
DeepEval scores your output with a *second* LLM. Configure it once per machine.

**Groq (cheap, free tier).** Groq speaks the OpenAI API, so register it as a "local model".
`deepeval set-grok` is xAI's Grok, not Groq.com, so do not use it.
```
deepeval set-local-model \
  --model openai/gpt-oss-120b \
  --base-url "https://api.groq.com/openai/v1" \
  --format json \
  --prompt-api-key
```

**OpenAI.**
```
export OPENAI_API_KEY=sk-...
deepeval set-openai --model gpt-4o-mini
```

Add `--save dotenv:.env.local` to either command to persist the key across sessions.
Switch back with `deepeval unset-local-model`.

05Threshold direction: the version trap

The chapter's second slide teaches that Hallucination, Toxicity and Bias are inverted: lower is better and the threshold is a ceiling. That is the DeepEval 3.x convention. The course's later notes record that DeepEval 4.x made every metric pass on score >= threshold, so 1.0 is always the good end, and that with 4.x installed, test_02's HallucinationMetric(threshold=0.3) is a weak floor rather than a 30% ceiling.

python -c "import inspect; from deepeval.metrics import BiasMetric; print(inspect.getsource(BiasMetric.is_successful))"

06Set up and run

From Notes.md and the course README. Python 3.11 or newer, one virtual environment, and a judge configured before the first run.

cd chapter_15_DeepEval
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -U deepeval requests
pip install openai python-dotenv   # test_02 imports both; Notes.md does not list them
python -c "import deepeval, requests; print(deepeval.__version__, requests.__version__)"

cp .env.sample .env                # set OPENAI_API_KEY and LOCAL_MODEL_API_KEY
pytest test_01_Anwser_Relevancy.py
pytest test_02_Groq_Qwen_Vs_GROQ_OpenAI.py
deepeval test run test_01_Anwser_Relevancy.py   # DeepEval's own runner, same test

The repo stores no run output for these two files. .env, .env.local, venv/ and .deepeval/ are gitignored; never commit a key.

07Gotchas

• test_02 calls Groq when it is imported. The question is asked at module level, so even pytest --collect-only runs that call: collecting the file needs a key and network access.
• A 200 OK is not an answer. While test_02 was being built, the configured model id was a prompt-guard classifier. It returned '0.0003637653135228902' with no error: a jailbreak probability, not a reply. Assert on the value, not the status.
• A variable name does not identify the provider. OPENAI_API_KEY held a Groq key. Read the prefix of the value, not the name of the variable.
• temperature=0 pins only the subject. The judge is still non-deterministic, so the same test can score differently on two runs.
• Different model families on purpose. qwen answers and gpt-oss grades, because "a judge scoring its own family inflates scores" (test_02's header).
• Relevancy is not correctness. A confidently wrong "5" is on topic. Add a metric that reads context or expected_output whenever the facts matter.
• Type the file name as it is. The first test is test_01_Anwser_Relevancy.py in the repo, spelling included.

DDrills for the chapter

Code drills use the two test files in chapter_15_DeepEval with a configured judge; judge scores and reasons vary from run to run, so the answers describe what holds, not exact numbers. Playwright drills target the live demo on this page (turn on Show locator badges to see every data-testid).

1. Relevancy versus truth

   In test_01 change actual_output to "5" and run. Does it fail? Which metric would catch it?

   Expected result

   Relevancy can still pass: "5" answers the question that was asked, and the course notes that a confidently wrong answer can score 1.0 here. Add HallucinationMetric with test_02's context, ["Basic arithmetic fact: 2 + 2 = 4"]. test_01's own context states no fact to contradict.
2. An off-topic reply

   Change actual_output to "Paris is the capital of France." and run test_01.

   Expected result

   Answer Relevancy drops far below 0.9 and the test fails with the judge's reason. The exact score and wording depend on the judge and the run.
3. Missing judge key

   Register Groq with set-local-model but leave LOCAL_MODEL_API_KEY unset, then run test_02.

   Expected result

   The subject call works and the judge fails with "Local API key is not configured", as recorded in the course's build notes.
4. Read the value, not the status

   Why is an HTTP 200 from the subject model not proof that the test is wired correctly?

   Expected result

   The prompt-guard classifier returned '0.0003637653135228902' with no error. Assert on the content (for example that the answer contains "4") before you trust any score built on it.
5. Which way does 0.3 point?

   With DeepEval 4.x installed, what does HallucinationMetric(threshold=0.3) in test_02 require?

   Expected result

   A score of at least 0.3. It is a weak floor, not "at most 30% hallucination". Making it strict means raising it, which the course leaves as the test author's call.
6. Count the cost

   How many paid calls does a three-metric test over 100 golden cases make, and why can the real bill be higher?

   Expected result

   300 by the course README's count, one per metric per case. Treat that as a floor: chapter 16 recorded three judge calls for a single Answer Relevancy case, so the number of requests can be several times higher.
7. Catch the wrong answer in the demoPlaywright

   Select the "confidently wrong 5" case, run it, and assert the status and both result rows.

   Hint

   Locators: de-case, de-run, de-status, and de-results with li[data-kind="ok"] and li[data-kind="bad"].
   Expected result

   de-status reads 1 failed: Hallucination did not meet its threshold. One PASS row (Answer Relevancy 1.00 >= 0.90) and one FAIL row (Hallucination 0.00 >= 0.30).
8. Trigger a missing-field errorPlaywright

   Load test_01, tick Faithfulness, and run. Assert that the run fails for the right reason.

   Hint

   The checkbox is de-m-faithfulness; test_01 has no retrieval_context, so de-use-retrieval is disabled.
   Expected result

   de-status contains Faithfulness needs retrieval_context and the results hold exactly one ERROR row.
9. Flip the pass rulePlaywright

   Run test_02, then choose the old slide rule with getByLabel('Pass rule') and run again.

   Expected result

   First 1 passed: every metric met its threshold., then 1 failed: Hallucination did not meet its threshold. with a note that the old rule flipped Hallucination.

SSolutions: the demo spec and the chapter's files

The Playwright spec passes against this page as written. The other tabs are the chapter's files, verbatim from the course repo.

import { test, expect } from '@playwright/test';

const URL = 'https://app.thetestingacademy.com/ai/blueprint/learn/deepeval.html';

test('relevancy passes a confidently wrong answer; hallucination fails it', async ({ page }) => {
  await page.goto(URL);
  await page.getByTestId('de-case').selectOption('wrong');
  await page.getByTestId('de-run').click();

  await expect(page.getByTestId('de-status')).toHaveText('1 failed: Hallucination did not meet its threshold.');
  const results = page.getByTestId('de-results');
  await expect(results.locator('li[data-kind="ok"]')).toContainText('Answer Relevancy 1.00 >= 0.90');
  await expect(results.locator('li[data-kind="bad"]')).toContainText('Hallucination 0.00 >= 0.30');
});

test('a metric errors when the field it reads is missing', async ({ page }) => {
  await page.goto(URL);
  await page.getByTestId('de-case').selectOption('t01');
  await page.getByTestId('de-m-faithfulness').check();
  await page.getByTestId('de-run').click();

  await expect(page.getByTestId('de-status')).toContainText('Faithfulness needs retrieval_context');
  await expect(page.getByTestId('de-results').locator('li[data-kind="bad"]')).toHaveCount(1);
});

test('reading a 4.x score with the old slide rule flips the verdict', async ({ page }) => {
  await page.goto(URL);
  await page.getByTestId('de-case').selectOption('t02');
  await page.getByTestId('de-run').click();
  await expect(page.getByTestId('de-status')).toHaveText('1 passed: every metric met its threshold.');

  await page.getByLabel('Pass rule').selectOption('slide');
  await page.getByTestId('de-run').click();
  await expect(page.getByTestId('de-status')).toContainText('1 failed: Hallucination did not meet its threshold.');
  await expect(page.getByTestId('de-status')).toContainText('flipped Hallucination');
});

#Flow:
# 1. Send "What is 2+2?" to Groq Qwen3.8-27b (the model under test).
# 2. Capture the raw answer.
# 3. Hand input + answer + context to DeepEval.
# 4. openai/gpt-oss-120b as judge -> scores AnswerRelevancy + Hallucination.
#    Subject != judge on purpose: a judge scoring its own family inflates scores.

GROQ_MODEL = "qwen/qwen3.8-27b"
JUDGE_MODEL = "openai/gpt-oss-120b"

import os

from dotenv import load_dotenv
from openai import OpenAI

from deepeval.metrics import AnswerRelevancyMetric, HallucinationMetric
from deepeval.test_case import LLMTestCase
from deepeval import assert_test

load_dotenv()

# Groq speaks the OpenAI API, so the official openai SDK works unchanged:
# just point base_url at Groq. The key in .env is a Groq key (gsk_...) that
# lives under OPENAI_API_KEY because that is the name deepeval's
# set-local-model path reads when talking to an OpenAI-compatible endpoint.
groq = OpenAI(
    api_key=os.getenv("OPENAI_API_KEY"),
    base_url="https://api.groq.com/openai/v1",
)


def llm_response(question):
    """Ask GROQ_MODEL one question and return its raw answer text."""
    response = groq.chat.completions.create(
        model=GROQ_MODEL,
        messages=[{"role": "user", "content": question}],
        # temperature=0 keeps the answer stable so the score below is
        # reproducible. The judge is still non-deterministic; this only
        # pins the thing under test.
        temperature=0,
    )
    return response.choices[0].message.content.strip()

question = "What is 2+2? Reply with just the number."
answer = llm_response(question)
print(f"\n[Groq {GROQ_MODEL}] → {answer!r}\n")



def test_l4_with_judge_qwen():
    case = LLMTestCase(
        input=question,
        actual_output=answer,
        expected_output="4",
        # HallucinationMetric scores actual_output against this grounding text.
        context=["Basic arithmetic fact: 2 + 2 = 4"],
    )

    metrics = [
        AnswerRelevancyMetric(threshold=0.8, model=JUDGE_MODEL),
        HallucinationMetric(threshold=0.3, model=JUDGE_MODEL),
    ]
    assert_test(case, metrics)

# Deepeval

### Pre Req.
* Python Installed
* pytest Basics knowledge
* LLM Brain - openai key or deepseek api key or groq api key, claude api ($5), 

* FREE LLM Brain
    * nvidia api free
    * amd api free (limited)

![alt text](image-1.png)

### Installation of Deepeval
* ```python3 -m venv venv```
* ```source venv/bin/activate```
* ```pip install --upgrade pip```
* ```pip install -U deepeval requests```

### Verify 
```
python -c "import deepeval, requests; print(deepeval.__version__, requests.__version__)"
```

![alt text](image.png)

**Deactivate later:**

```
deactivate
```

### Pick the judge LLM
DeepEval scores your output with a *second* LLM. Configure it once per machine.

**Groq (cheap, free tier).** Groq speaks the OpenAI API, so register it as a "local model".
`deepeval set-grok` is xAI's Grok, not Groq.com, so do not use it.
```
deepeval set-local-model \
  --model openai/gpt-oss-120b \
  --base-url "https://api.groq.com/openai/v1" \
  --format json \
  --prompt-api-key
```

**OpenAI.**
```
export OPENAI_API_KEY=sk-...
deepeval set-openai --model gpt-4o-mini
```

Add `--save dotenv:.env.local` to either command to persist the key across sessions.
Switch back with `deepeval unset-local-model`.

### Run
```
pytest test_01_Anwser_Relevancy.py
```

### Watch out
* Every metric assertion is a real, paid LLM call. Keep the golden dataset small.
* `.env`, `.env.local`, `venv/` and `.deepeval/` are gitignored. Never commit a key.