The Testing Academy · Class Notes Thursday, 12 November (IST)
Live class · study guide

AI agents in Playwright: an LLM gateway and AI test data

Where AI fits in a Playwright framework: open-source or closed-source models and when to use each, free models on Groq and Ollama, an AI agent as brain, memory and tools, then LLMGateway.ts, one entry point for five LLM providers, and a checkout test whose customer now comes from an AI prompt instead of Faker.

By Pramod Dutta, The Testing Academy. Study notes from the Playwright 3x class, built from the session recording, the batch repo and the 3x Eraser notes deck. The code is AdvancePlaywrightFramework3x at commit "feat: add multi-provider LLM gateway and AI test data", pushed to the llm-aiagents branch as the class ended and merged into main a minute later; it was still the newest when checked at 09:24 IST on 12 November. The deck's 12 November section holds the model notes, the Groq request, the three parts of an AI agent, the two prompts used in class and the next class's agents. The class's own run of the checkout spec, at 07:57, was read from the test results in the instructor's local folder: both tests passed, with DeepSeek's deepseek-chat. No API key was used for these notes and nothing signed in anywhere. The checkout spec signs in to TTACart, so it was not run. The gateway spec ran as pushed, with no key: one test passed and one skipped. Copies of the gateway spec and of DataGenerator.ts, each with one line changed, ran against a local model, gpt-oss:20b on Ollama, on a local machine. The provider choice, the checkout method's checks and the deck's request body were run offline.

Hides every output until you tap Reveal, so you can predict it first.
01

What this class covered

  • Playwright's AI tools, and agents inside the framework
  • Open-source or closed-source models
  • An LLM client, and free models on Groq and Ollama
  • An AI agent: brain, memory and tools
  • Test data from AI instead of Faker
  • LLMGateway.ts: one entry point for five providers
  • The checkout test, with its customer from AI
  • Tasks, and what comes next
  • Use an open-source model for a task in bulk, and a closed-source one when the task must be specific or accurate, such as code generation.
  • An AI agent has a brain, memory and tools. The class's first agents are not fully agents yet: they become agents when they use memory and tools.
  • AI agents can sit before a Playwright run, to generate the test data, or after it, to work on the results: RCA, flaky tests, healing.
  • Use AI for test data Faker cannot make, data specific to your company and close to real data, and ask for JSON only.
  • LLMGateway is one entry point: the code, then LLM_PROVIDER, then the first key found decide the provider, and LLM_ALLOWED_PROVIDERS refuses any other.
  • Keep the prompt in a file, and vary it on every call: the class's prompt gets random initials and a state from Faker each time.
  • No key? The gateway spec skips its LLM test. Groq has a free model, and Ollama runs one on your own machine for free.
02

Playwright's AI tools, and agents inside the framework

PLAYWRIGHT'S AI TOOLS, AND THE AGENTS INSIDE THE FRAMEWORK Covered in the recordings: Playwright's own AI tools PLAYWRIGHT MCP Generates a test case or helps you build one PLAYWRIGHT AI AGENTS Planner, generator, healer planner: plans the app generator: it's fine healer: very bad PLAYWRIGHT CLI An alternate to MCP a much cheaper version This class and the next: AI agents inside the framework BEFORE THE RUN AI test data this class: an LLM gateway and a prompt file THE RUN Playwright tests the framework's specs AFTER THE RUN Agents on the results next class: an RCA agent, a flaky test finder, a self-healer An AI agent has a brain, memory and tools, and it can sit before a run or after it.
The tools in the recordings are Playwright's own. This class starts on agents that run inside the framework, before a run; the next class adds the ones that work on the results.

Already in the recordings. Before the agents, the instructor listed three things the course has covered in recorded sessions. Playwright MCP helps you generate a test case, or build one. Playwright's AI agents are a test planner, a generator and a healer. The planner plans the app and generates the code later, the generator is fine, and the healer, to be honest, is very bad: it can help you heal a test, but it is not good. Playwright CLI is an alternate to MCP, a much cheaper version. The TA was asked to find the three recordings and pin them in the group, if possible, and everyone was asked to take out this Saturday and Sunday to watch them.

Today: agents inside the framework. The last class closed the API part of the framework. This one showed something different, the instructor said: how to add independent AI agents into the Playwright framework itself. The deck's 12 November section lists the same three tools with the same verdicts and, further on, the plan for the framework: pre and post Playwright agents.

03

Open-source or closed-source models

Remember, the instructor said: large language models come in two kinds, open source and closed source. The rule the class gave: when you want to do a task in bulk, go with open source. When you want a task done specifically, or more accurately, use closed source.

The instructor's own examples:

Model named in class Used for, in the instructor's examples
Mistral, open source OCR, PDF to text: more than 3,000 of almost 12,000 test cases verify a PDF's text, which would be very costly with a closed-source model. Also some basic AI agents.
Claude Opus 5.5, closed source Code generation, which always gets a closed-source model. Also some specific agentic QA.
DeepSeek V4 Flash Bulk operations, much cheaper, installed locally in some companies: RCA agents and a flaky test case finder.
GPT-5 Mini Small things, like RCA.

Mistral is a European open-source model, the instructor added. The deck's note on these models has the same split: open source for a task in bulk, with the OCR and the DeepSeek examples, and closed source for a specific or accurate task, with Claude Opus 5.5 for code generation.

OpenRouter, a curator. OpenRouter is a curator, the instructor said: through it you can use open-source and closed-source models, both of them, and you can select free ones. The instructor's example of what it costs: a credit of about $500 a month, somewhere around $500 to $1,000 the instructor thought, added to OpenRouter and used on some part of Mistral, some part of Opus 5.5 or some part of GPT-5 Mini. The deck lists the same three under OpenRouter.

04

An LLM client, and free models on Groq and Ollama

To bring AI into your automation framework, you need to add an LLM client, the instructor said. Most LLM clients follow the same OpenAI API specification, and OpenRouter's API, for example, follows the same thing. The deck lists OpenAI's chat completions API: POST, GET and DELETE on /v1/chat/completions.

The client takes an input, a model and data, makes a POST request, and returns an output, which the automation code then uses: that is the deck's note on the LLM gateway. Call it an LLM gateway or a client, as you like. The instructor generally calls it a client, because it helps you make a request and return an output.

Groq, free. Apart from OpenRouter, there is a very famous website that is free of cost: groq.com, spelled G-R-O-Q. Create a free account and use it. The instructor generally prefers a free open-source model, and Groq has one: OpenAI's open-source GPT, the 120 billion one, openai/gpt-oss-120b in the deck. To be honest, the instructor said, it is one of the best models, and it can do most of your things.

The request, as the deck has it: a POST to Groq's URL, with your key and a question.

Terminal
curl "https://api.groq.com/openai/v1/chat/completions" \
-X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${GROQ_API_KEY}" \
-d '{
       "messages": [
         {
           "role": "user",
           "content": "what is 2+2?"
         },
        ],
       "model": "openai/gpt-oss-120b",
       "temperature": 1,
       "max_completion_tokens": 2048,
       "top_p": 1,
       "stream": true,
       "reasoning_effort": "medium",
       "stop": null
     }'

The key comes from the GROQ_API_KEY environment variable, so it is never typed into the command.

One comma too many. In the deck's request body, a comma follows the only message, just before the ]. JSON allows no comma after the last item, so a strict parser rejects the body as written. Remove that comma before you send it.

Checked for these notes, offline, with Node's JSON.parse:

JavaScript
// The -d body of the deck's Groq curl, exactly as written, then with the comma after the message removed.
const asWritten = `{
       "messages": [
         {
           "role": "user",
           "content": "what is 2+2?"
         },
        ],
       "model": "openai/gpt-oss-120b",
       "temperature": 1,
       "max_completion_tokens": 2048,
       "top_p": 1,
       "stream": true,
       "reasoning_effort": "medium",
       "stop": null
     }`;
const commaRemoved = asWritten.replace('},\n        ]', '}\n        ]');

for (const [name, body] of [['as written', asWritten], ['comma removed', commaRemoved]]) {
  try {
    JSON.parse(body);
    console.log(`${name}: valid JSON`);
  } catch (e) {
    console.log(`${name}: ${e.message}`);
  }
}
Output
as written: Unexpected token ']', ...",
        ],
       "... is not valid JSON
comma removed: valid JSON

What can you ask? Anything, the instructor said: your logs, your failures, your automation results. After Playwright has run, you can pass the results and ask what the output is, or what the failure count is.

Ollama, free and local. If you have access to Ollama, in your company or on a machine where it runs, you can ask it the same way: it is 100% free, and you can make an unlimited number of requests. It is available as an API, so a curl request works too, and it can be used in your automation. Ollama is free, the instructor confirmed in answer to the chat, and LM Studio can be used too; there you use open-source models only. Every model reply in these notes was made that way, with Ollama on a local machine and no key, as the LLMGateway section shows.

05

An AI agent: brain, memory and tools

Two panels. Left, titled Brain, with the sublabel Not fully agents: a simple stylised brain on a round plinth, with one speech bubble of grey squiggles on each side and nothing else connected. Right, titled Brain, memory, tools, with the sublabel An AI agent: the same brain on its plinth, joined by short lines to an open notebook with a ribbon bookmark and an open toolbox holding a wrench, a screwdriver and a gear, the same bubbles on each side, and a terracotta tick above the answer.
The instructor's definition: an AI agent has a brain, memory and tools. The class's first agents are not fully agents yet; they become agents when they use memory and tools.

Before going further, remember this, the instructor said: an AI agent has a brain, it has memory, and it has access to tools. The deck lists the same three. The agents built in this class are called agents, but to be honest, they are not fully agents yet: they become agents when they start using memory as well as tools.

An agent can be an independent system, a part of your framework, or a separate file. And it can sit anywhere. Before the automation runs, it can generate the test data with AI. After the run, it can call the LLM client with the results and the logs, say there are failures, and ask why. You stick these calls together, before or after the Playwright run, and let them do their task. The deck's heading for this is "Pre and Post Playwright Agents".

06

Test data from AI instead of Faker

The first agent the class adds is a data generator. The framework makes its test data with FakerJS, as in the data-driven class. What if you do not want to use Faker? Suppose Faker is not allowed in your company, or you want data specific to your company: Faker always creates fake data, which looks similar.

The instructor's example: test data for cars. Every car has a VIN number and a chassis number, and there are internal numbers such as the engine number, all of which should be very close to real data. That is very difficult with FakerJS. Another of the instructor's examples: a very specific customer detail, or a very specific bank account number.

The plan, as the class put it: create a structured file that says how you want the data, make a request to the LLM client through the LLM gateway, and it generates the data, according to your project, for the test cases to use. The deck draws it as a structure file in Markdown, then the LLMGateway, then the generated data, in a file it calls AIDataGenutil.ts. In the repo, that last step became a method of DataGenerator, in the checkout section below.

In Groq's playground. The instructor asked for close-to-real data for an Indian car, a Hyundai i10, in a given format, and got it back: a VIN for a car from India, a Hyundai i10 hatchback. Then the prompt was made more structured, so that the model always replies with only the JSON, no extra things, because that JSON is what the automation consumes. The model was OpenAI's open-source GPT, the 120 billion version, with a maximum number of tokens and a user prompt, and an API key is required, of course.

The same on Ollama. The instructor then showed the same kind of request answered by Ollama: 100% free, with unlimited requests, if you have a machine where Ollama is running. Asked whether it will generate a credit card or IBAN number, the kind net banking uses: absolutely yes. The instructor asked for one, in JSON format only, and got it: data specific to your company, where FakerJS is not useful.

07

LLMGateway.ts: one entry point for five providers

LLMGATEWAY.TS: ONE ENTRY POINT, FIVE PROVIDERS, TWO REQUEST SHAPES A TEST OR A UTIL Asks for JSON llm.askJson(prompt) or ask() for text LLMGATEWAY Picks the provider 1. new LLMGateway({ provider }) 2. LLM_PROVIDER in .env 3. the first provider with a key LLM_ALLOWED_PROVIDERS refuses any other provider OPENAI SHAPE /chat/completions Groq, OpenRouter, OpenAI, DeepSeek ANTHROPIC SHAPE /v1/messages Claude THE REPLY extractJson() the JSON, even inside a fence or with chatter around it BACK IN THE TEST Parsed JSON res.data or a file, with saveTo either reply Five providers, two request shapes: the test asks once, and the code or .env decides which provider answers.
src/ai/LLMGateway.ts at the class commit, drawn as a flow. The order of choice is its header comment, lines 11 to 17; the two shapes are its two request builders.

A new branch. Back in the framework, the class worked on a new branch, to be merged later. If you are lazy, you can ask the AI to create it, but why waste your tokens? git checkout -b does it. In the repo the branch is llm-aiagents, merged into main as the class ended.

The prompt. The instructor gave the AI coding agent one prompt, and said it would be shared with the class. The deck holds it:

there is an AI folder that we have created in the source directory. I want you to do one thing: I want you to add an LLM gateway for this. The LLM gateway is a TS file or a utility file that is available in the AI folder. It will provide us access to groq.com, OpenRouter, OpenAI, Claude AI (all free), plus DeepSeek direct API access also. Depending upon my company: - If my company allows using OpenRouter, I will use OpenRouter. - If my company allows using groq.com, I will use that one. - If my company allows OpenAI, I will use OpenAI. - If my company allows direct Anthropic access to Claude, I will use that also, right? Create multiple LLM gateway access that will take a prompt and give us an output as a JSON file or JSON as a response directly.

What the agent built. src/ai/LLMGateway.ts. The beauty of it, the instructor said, is that it has one entry point: whichever provider your company is using, you just tell it which provider you want, and it uses that one. It is totally vibe-coded: the instructor did not code anything in it. Its header says how it works:

LLMGateway.tsTypeScript
/**
 * LLMGateway: one entry point for every LLM provider your company allows.
 *
 * Supported providers (pick whichever your company has approved):
 *   - groq        -> https://api.groq.com          (free tier, OpenAI-compatible)
 *   - openrouter  -> https://openrouter.ai         (free ":free" models, OpenAI-compatible)
 *   - openai      -> https://api.openai.com
 *   - anthropic   -> https://api.anthropic.com     (Claude, Messages API)
 *   - deepseek    -> https://api.deepseek.com      (direct API, OpenAI-compatible)
 *
 * Choosing a provider (first match wins):
 *   1. new LLMGateway({ provider: 'groq' })      explicit in code
 *   2. LLM_PROVIDER=groq                         in .env
 *   3. the first provider that has an API key, in the order of PROVIDER_ORDER
 *
 * Company policy: set LLM_ALLOWED_PROVIDERS=groq,openrouter and the gateway
 * refuses to call anything outside that list, even if a key for it is set.
 *
 * Usage:
 *   const llm = new LLMGateway();                           // auto from .env
 *   const { data } = await llm.askJson<{ cases: string[] }>(
 *       'Give 3 negative test cases for a login form as {"cases": [...]}',
 *   );
 *   await llm.askJsonToFile('...prompt...', 'reports/ai/login-cases.json');
 *   const text = await llm.ask('Explain this error: ...');   // plain text
 *
 * Uses Node's built-in fetch, so it needs no extra npm packages.
 */

The only thing to be concerned about, the instructor said, is the model. Each provider has a default model, and you can change the model when you make a request. Here is the provider registry, lines 124 to 170, with the cost column from the README:

Provider Request shape JSON mode Default model Cost, from the README
Groq, groq OpenAI yes llama-3.3-70b-versatile free tier
OpenRouter, openrouter OpenAI no meta-llama/llama-3.3-70b-instruct:free free :free models
DeepSeek, deepseek OpenAI yes deepseek-chat paid, low cost
OpenAI, openai OpenAI yes gpt-4o-mini paid
Anthropic, anthropic Anthropic no claude-haiku-5-5 paid

The prompt calls them all free. The README's table is more exact: Groq has a free tier and OpenRouter has free :free models, while DeepSeek is paid, at low cost, and OpenAI and Anthropic are paid. The deck's free Groq model is not the gateway's Groq default either. Line 121 shows how to pick it: GROQ_MODEL=openai/gpt-oss-120b in .env.

Groq, OpenRouter, OpenAI and DeepSeek take the same request shape, OpenAI's chat completions, and Claude takes Anthropic's messages API. So the gateway needs two request builders, not five clients. The repo's learnings note, which the agent also wrote, starts there: count the wire formats.

The keys go in .env. The environment is where your OpenRouter key or DeepSeek key goes, the instructor thought, whichever you want to use. The commit added this block to .env.example:

.env.exampleTerminal
# ---- LLM Gateway (src/ai/LLMGateway.ts) ----
# Pick the provider your company allows: groq | openrouter | openai | anthropic | deepseek
# Leave empty to auto-pick the first provider that has a key (groq, openrouter, deepseek, openai, anthropic).
LLM_PROVIDER=
# Company allowlist. The gateway refuses any provider not listed here. Empty = all allowed.
# LLM_ALLOWED_PROVIDERS=groq,openrouter
# LLM_TIMEOUT_MS=60000
# LLM_MAX_RETRIES=2

# Keys (only the ones you use). Models are optional overrides of the built-in defaults.
# GROQ_API_KEY=            # free: https://console.groq.com/keys
# GROQ_MODEL=llama-3.3-70b-versatile
# OPENROUTER_API_KEY=      # free ":free" models: https://openrouter.ai/keys
# OPENROUTER_MODEL=meta-llama/llama-3.3-70b-instruct:free
# OPENAI_MODEL=gpt-4o-mini
# ANTHROPIC_MODEL=claude-haiku-5-5
# DEEPSEEK_API_KEY=        # https://platform.deepseek.com/api_keys
# DEEPSEEK_MODEL=deepseek-chat

LLM_ALLOWED_PROVIDERS is the company allowlist: as the header's lines 16 and 17 say, the gateway refuses anything outside the list, even when a key for it is set.

The agents folder. src/ai/agents/ already held an RCA agent and a flaky analyzer, and the instructor wondered how the flaky test case analyzer got there: maybe the AI copied them from earlier work, the instructor guessed, and set them aside for now. In the repo they came with the framework commit of 20 October, as helpers for the TTA report.

The spec. The agent also wrote llm-gateway.spec.ts, a basic file to test that the results come back. It asks for three negative test cases for a login form:

llm-gateway.spec.tsTypeScript
import { test, expect } from '@playwright/test';
import { LLMGateway, availableProviders, extractJson } from '../../ai/LLMGateway';

/**
 * LLM Gateway demo. Uses whichever provider .env selects (LLM_PROVIDER or the
 * first key found). Skips cleanly when no LLM key is configured.
 *
 *   npx playwright test src/tests/aiTests --project=chromium
 *   LLM_PROVIDER=groq npx playwright test src/tests/aiTests --project=chromium
 */

interface LoginCases {
    cases: { title: string; username: string; password: string; expected: string }[];
}

test.describe('LLM Gateway', () => {
    test('extractJson tolerates fences and chatter', () => {
        expect(extractJson('```json\n{"a":1}\n```')).toEqual({ a: 1 });
        expect(extractJson('Sure! Here it is: {"b":[1,2]} Hope that helps.')).toEqual({ b: [1, 2] });
        expect(extractJson('[{"c":"}"}]')).toEqual([{ c: '}' }]);
    });

    test('generates login test data as JSON and saves it', async () => {
        test.skip(availableProviders().length === 0 && !process.env.LLM_PROVIDER, 'No LLM API key in .env');
        test.setTimeout(120_000);

        const llm = new LLMGateway();
        const res = await llm.askJson<LoginCases>(
            'Generate 3 negative test cases for a login form. Return ' +
                '{"cases":[{"title":string,"username":string,"password":string,"expected":string}]}',
            { system: 'You are a senior QA engineer.', temperature: 0, saveTo: 'reports/ai/login-cases.json' },
        );

        console.log(`${res.provider}/${res.model} in ${res.latencyMs}ms`, res.data);
        expect(Array.isArray(res.data.cases)).toBeTruthy();
        expect(res.data.cases.length).toBeGreaterThan(0);
        expect(res.savedTo).toContain('login-cases.json');
    });
});

Run for these notes as pushed, with no key: the first test passes, and the second skips itself, as line 24 says it should. The first test is the gateway's JSON parser at work: extractJson() finds the JSON inside a json code fence, inside a sentence of chatter, and with a } inside a string. The README puts the whole of askJson() this way: it asks for JSON only, uses the provider's JSON mode where it is reliable, then pulls the first valid JSON block out of the reply, and if there is none, it throws LLMGatewayError with the raw reply attached.

Terminal
npx playwright test src/tests/aiTests/llm-gateway.spec.ts --reporter=list --workers=1
Output
Running 2 tests using 1 worker

  ✓  1 [chromium] › src/tests/aiTests/llm-gateway.spec.ts:17:9 › LLM Gateway › extractJson tolerates fences and chatter (5ms)
  -  2 [chromium] › src/tests/aiTests/llm-gateway.spec.ts:23:9 › LLM Gateway › generates login test data as JSON and saves it

  1 skipped
  1 passed (469ms)

Which provider? Run for these notes, offline, with no key set anywhere: what the gateway picks, or the error it gives, for a few settings.

TypeScript
import { test } from '@playwright/test';
import { LLMGateway, resolveProvider, availableProviders, type LLMProvider } from '../../ai/LLMGateway';

// Which provider does the gateway pick? Each row sets the environment, asks, and prints the answer or the error.
// No API key is set in any row, and nothing is sent anywhere.
const ROWS: [string, Record<string, string>, LLMProvider?][] = [
    ['nothing set', {}],
    ['LLM_PROVIDER=deepseek', { LLM_PROVIDER: 'deepseek' }],
    ["LLM_PROVIDER=deepseek, and the code asks for 'groq'", { LLM_PROVIDER: 'deepseek' }, 'groq'],
    ['LLM_PROVIDER=ollama', { LLM_PROVIDER: 'ollama' }],
    ['LLM_ALLOWED_PROVIDERS=groq,openrouter, LLM_PROVIDER=deepseek', { LLM_ALLOWED_PROVIDERS: 'groq,openrouter', LLM_PROVIDER: 'deepseek' }],
    ['LLM_ALLOWED_PROVIDERS=groq,openrouter, nothing else', { LLM_ALLOWED_PROVIDERS: 'groq,openrouter' }],
];

function setEnv(env: Record<string, string>) {
    delete process.env.LLM_PROVIDER;
    delete process.env.LLM_ALLOWED_PROVIDERS;
    Object.assign(process.env, env);
}

test('which provider the gateway picks', () => {
    console.log(`keys found: [${availableProviders().join(', ')}]`);
    for (const [label, env, explicit] of ROWS) {
        setEnv(env);
        try {
            console.log(`${label}\n  -> ${resolveProvider(explicit)}`);
        } catch (e) {
            console.log(`${label}\n  error: ${(e as Error).message}`);
        }
    }
    setEnv({ LLM_PROVIDER: 'deepseek' });
    try {
        new LLMGateway();
    } catch (e) {
        console.log(`new LLMGateway() with LLM_PROVIDER=deepseek\n  error: ${(e as Error).message}`);
    }
});
Output
Running 1 test using 1 worker

keys found: []
nothing set
  error: No LLM provider configured. Set one of GROQ_API_KEY, OPENROUTER_API_KEY, DEEPSEEK_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY in .env (and optionally LLM_PROVIDER).
LLM_PROVIDER=deepseek
  -> deepseek
LLM_PROVIDER=deepseek, and the code asks for 'groq'
  -> groq
LLM_PROVIDER=ollama
  error: Unknown LLM provider "ollama". Valid: groq, openrouter, openai, anthropic, deepseek
LLM_ALLOWED_PROVIDERS=groq,openrouter, LLM_PROVIDER=deepseek
  error: Provider "deepseek" is not in LLM_ALLOWED_PROVIDERS (groq, openrouter).
LLM_ALLOWED_PROVIDERS=groq,openrouter, nothing else
  error: No LLM provider configured. Set one of GROQ_API_KEY, OPENROUTER_API_KEY in .env (and optionally LLM_PROVIDER).
new LLMGateway() with LLM_PROVIDER=deepseek
  error: DEEPSEEK_API_KEY is not set. Add it to .env to use provider "deepseek".
  ✓  1 [chromium] › src/tests/aiTests/resolve.spec.ts:21:5 › which provider the gateway picks (1ms)

  1 passed (318ms)

The code comes first, then LLM_PROVIDER, then the first provider that has a key, and the allowlist refuses the rest. A provider without its key stops at the constructor, line 312. That is also what the spec's second test meets when LLM_PROVIDER is set without its key: line 24 skips the test only when there is no key and no LLM_PROVIDER either. Ollama is not one of the five names.

With a local model, and no key. The gateway's baseUrl option, lines 49 and 50, is there for a proxy or a self-hosted mirror, and Ollama answers OpenAI's request shape on your own machine. So for these notes a copy of the spec, llm-gateway.local.spec.ts, changes line 27 and nothing else:

TypeScript
        const llm = new LLMGateway({ provider: 'openai', baseUrl: 'http://127.0.0.1:11434/v1', apiKey: 'ollama', model: 'gpt-oss:20b' });

Ollama needs no key, but the gateway will not start without one (line 312), so any word fills the slot. The model is gpt-oss:20b, a smaller size of the same gpt-oss model the deck names for Groq: 20.9 billion parameters, as Ollama reports it. Run with LLM_PROVIDER set, so that line 24 does not skip the test:

Terminal
LLM_PROVIDER=openai npx playwright test src/tests/aiTests/llm-gateway.local.spec.ts --reporter=list --workers=1
Output
Running 2 tests using 1 worker

  ✓  1 [chromium] › src/tests/aiTests/llm-gateway.local.spec.ts:17:9 › LLM Gateway › extractJson tolerates fences and chatter (4ms)
09:00:13 [info] [LLMGateway] -> openai/gpt-oss:20b (138 chars, json)
09:00:19 [info] [LLMGateway] <- openai/gpt-oss:20b 6254ms (in 430 / out 85 tokens)
09:00:19 [info] [LLMGateway] saved JSON -> reports/ai/login-cases.json
openai/gpt-oss:20b in 6254ms {
  cases: [
    {
      title: 'Empty username',
      username: '',
      password: 'somePassword123',
      expected: 'Error: Username cannot be empty.'
    },
    {
      title: 'Wrong password',
      username: 'validUser',
      password: 'wrongPass!',
      expected: 'Error: Invalid credentials.'
    },
    {
      title: 'SQL injection attempt',
      username: "' OR '1'='1",
      password: "' OR '1'='1",
      expected: 'Error: Invalid credentials.'
    }
  ]
}
  ✓  2 [chromium] › src/tests/aiTests/llm-gateway.local.spec.ts:23:9 › LLM Gateway › generates login test data as JSON and saves it (6.3s)

  2 passed (6.7s)

The logger's dates are removed. Every request the run made was logged, and there was one: a POST to Ollama's /v1/chat/completions on 127.0.0.1. The same three cases went to reports/ai/login-cases.json, as line 31 asks.

08

The checkout test, with its customer from AI

THE CHECKOUT TEST'S CUSTOMER: FROM A PROMPT, NOT FROM FAKER THE PROMPT FILE One customer, three rules src/ai/prompts/ checkoutCustomer.prompt.md FAKER, BEFORE EACH CALL Fills the placeholders {{FIRST_INITIAL}} {{LAST_INITIAL}} {{STATE}} LLMGATEWAY askJson(prompt) temperature 1 in class: DeepSeek THE CHECK Three fields, or throw firstName, lastName, postalCode THE SPEC, LINE 38 fillGuest(customer) on checkout step one BEFORE THIS CLASS checkoutCustomer() Faker's first name, last name and postal code replaced by checkoutCustomerAI() Edit the prompt: it says what the customer looks like. The code checks the three fields the form needs. Random initials and a state go into every prompt, so the calls stop returning the same customer.
DataGenerator.checkoutCustomerAI(), lines 130 to 144 of src/utils/DataGenerator.ts, with the spec line that calls it and the Faker method it replaced.

The prompt. Then an end-to-end test. After the Ponytail review, the e2e checkout spec builds its customer with DataGenerator.checkoutCustomer(), which Faker fills. Now the data was to be generated via AI, and wiring is important here, the instructor said. The second prompt, also in the deck:

in this file, replace the DataGenerator.checkoutCustomer(), to the AI Gnerated Part, Make the prompt avaiable outside in the /src/ai/prompts folder for the datageneration, you can add a extra method which will use the LLMGateway and create the data and run it to verify, I am adding the key for the deepseek api in the env variable, make a ponytail review and minium code changes.

For the run, the instructor created a DeepSeek key on screen and added it to the environment, said it would be deleted and not to use it, and later in the class said it would be deleted.

One line changed. The new spec, src/tests/aiTests/e2e-checkout_DataGenAI.spec.ts, is the reviewed spec with one line different, line 38:

e2e-checkout_DataGenAI.spec.tsTypeScript
        await visualStep(page, 'Checkout step one: enter customer details', async () => {
            await cartPage.checkout();
            await checkoutStepOnePage.assertLoaded();
            await checkoutStepOnePage.fillGuest(await DataGenerator.checkoutCustomerAI());
        });

In e2e-checkout_validFixture_PonyTail_Review.spec.ts the same line passes DataGenerator.checkoutCustomer(). Now the function name has changed, as the instructor pointed out, to checkoutCustomerAI(), in DataGenerator.ts:

DataGenerator.tsTypeScript
    /** Customer info for the TTACart checkout step-one form. */
    static checkoutCustomer(): CheckoutCustomer {
        return {
            firstName: DataGenerator.firstName(),
            lastName: DataGenerator.lastName(),
            postalCode: DataGenerator.postalCode(),
        };
    }

    /**
     * Customer info generated by an LLM (provider from .env, see src/ai/LLMGateway.ts).
     * The prompt lives in src/ai/prompts/checkoutCustomer.prompt.md.
     */
    static async checkoutCustomerAI(): Promise<CheckoutCustomer> {
        // Random placeholders stop the model returning the same person on every call.
        const prompt = fs
            .readFileSync(path.resolve(__dirname, '../ai/prompts/checkoutCustomer.prompt.md'), 'utf8')
            .replace('{{FIRST_INITIAL}}', faker.string.alpha({ casing: 'upper' }))
            .replace('{{LAST_INITIAL}}', faker.string.alpha({ casing: 'upper' }))
            .replace('{{STATE}}', faker.location.state());
        const { data } = await new LLMGateway().askJson<CheckoutCustomer>(prompt, { temperature: 1 });
        for (const key of ['firstName', 'lastName', 'postalCode'] as const) {
            if (typeof data?.[key] !== 'string' || !data[key].trim()) {
                throw new Error(`[DataGenerator] AI customer is missing "${key}": ${JSON.stringify(data)}`);
            }
        }
        return { firstName: data.firstName, lastName: data.lastName, postalCode: data.postalCode };
    }

Lines 132 to 136 read the prompt from src/ai/prompts and fill its three placeholders with Faker. Line 137 asks the gateway for JSON at temperature 1. Lines 138 to 142 throw if any of the three fields is missing or empty. This is how the first name, last name and postal code are created now, the instructor said: by AI, from this prompt.

checkoutCustomer.prompt.mdText
Generate one realistic, unique customer for an e-commerce checkout form.

Rules:
- firstName: a real-looking first name starting with "{{FIRST_INITIAL}}", letters only, 2-20 characters.
- lastName: a real-looking last name starting with "{{LAST_INITIAL}}", letters only, 2-20 characters.
- postalCode: a valid-looking ZIP code in {{STATE}}, exactly 5 digits, as a string.
- Avoid common placeholder names like John Doe.

Return exactly this JSON shape and nothing else:
{"firstName": "string", "lastName": "string", "postalCode": "string"}

The class's run. The instructor had the agent run the spec and open the report: the first name and last name were now created by AI. That run, at 07:57, is in the test results left in the instructor's local folder. Both tests passed: the checkout, and the check that a locked-out user is refused. The trace keeps the gateway's two log lines, here with the date removed:

Text
07:57:47 [info] [LLMGateway] -> deepseek/deepseek-chat (508 chars, json)
07:57:48 [info] [LLMGateway] <- deepseek/deepseek-chat 1004ms (in 168 / out 25 tokens)

The customer it typed into the form was Vanessa Galloway, with the ZIP code 40203.

Change the prompt. The beauty of it, the instructor said: you only have to create a prompt. If you want something different, you change the prompt, and the rest of the things remain the same. So this AI agent generates the data response using an LLM client, the instructor said: absolutely yes.

The prompt changes the values, not the fields. The prompt decides what the customer looks like: the names, their style, the ZIP code's state. The fields are fixed in code: lines 138 to 143 check and return exactly firstName, lastName and postalCode, the three that the checkout form takes. To get other fields, the method and the form code must change too. Ask the model for other fields and nothing else, and checkoutCustomerAI() throws, as the second check below shows.

Run for these notes, offline: the class's method with no key at all, then a copy whose gateway gets a fixed reply with other fields, so no model is called.

TypeScript
import { test } from '@playwright/test';
import { LLMGateway } from '../../ai/LLMGateway';
import { DataGenerator } from '../../utils/DataGenerator';
import { DataGenerator as LocalDataGenerator } from '../../utils/DataGenerator.local';

test('checkoutCustomerAI with no key, and with a reply in other fields', async () => {
    // 1. The class's method as pushed, with no key in the environment.
    try {
        await DataGenerator.checkoutCustomerAI();
    } catch (e) {
        console.log(`no key:\n  ${(e as Error).message}`);
    }

    // 2. A prompt asking for other fields: the model's reply replaced by a fixed one, so no model is called.
    LLMGateway.prototype.askJson = async function () {
        return { data: { fullName: 'Sample Customer', pinCode: '400001' } } as never;
    };
    try {
        await LocalDataGenerator.checkoutCustomerAI();
    } catch (e) {
        console.log(`a reply with other fields:\n  ${(e as Error).message}`);
    }
});
Output
Running 1 test using 1 worker

no key:
  No LLM provider configured. Set one of GROQ_API_KEY, OPENROUTER_API_KEY, DEEPSEEK_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY in .env (and optionally LLM_PROVIDER).
a reply with other fields:
  [DataGenerator] AI customer is missing "firstName": {"fullName":"Sample Customer","pinCode":"400001"}
  ✓  1 [chromium] › src/tests/aiTests/checkout-ai.checks.spec.ts:6:5 › checkoutCustomerAI with no key, and with a reply in other fields (1ms)

  1 passed (482ms)

The first error is what the checkout spec meets at checkout step one when there is no key. The README warns about the same: the checkout test now fails if the LLM is unreachable, so use DataGenerator.checkoutCustomer() where that is not acceptable.

Why the placeholders. The README's gotcha: the same prompt gives the same answer, even at temperature 1, and three calls once returned "Marisol, 63108" every time. So the prompt has {{FIRST_INITIAL}}, {{LAST_INITIAL}} and {{STATE}}, which Faker fills before each call. Run for these notes against the local model, through a copy of DataGenerator.ts whose line 137 passes the local options shown above, three calls in a row:

TypeScript
import { test } from '@playwright/test';
import { LLMGateway } from '../../ai/LLMGateway';
import { DataGenerator } from '../../utils/DataGenerator.local';

// DataGenerator.local.ts is the class's DataGenerator.ts with line 137 changed, so its gateway calls a local model.
// This wrapper prints what each filled-in prompt asked for, then lets the call go ahead.
const askJson = LLMGateway.prototype.askJson;
LLMGateway.prototype.askJson = function (this: LLMGateway, prompt: string, opts?: object) {
    const first = prompt.match(/first name starting with "(\w)"/)?.[1];
    const last = prompt.match(/last name starting with "(\w)"/)?.[1];
    const state = prompt.match(/ZIP code in (.+?), exactly/)?.[1];
    console.log(`asked: first name ${first}..., last name ${last}..., a ZIP code in ${state}`);
    return askJson.call(this, prompt, opts);
} as typeof askJson;

test('three AI customers from a local model', async () => {
    test.setTimeout(300_000);
    for (let call = 1; call <= 3; call++) {
        const customer = await DataGenerator.checkoutCustomerAI();
        console.log(`got:   ${JSON.stringify(customer)}`);
    }
});
Output
Running 1 test using 1 worker

asked: first name P..., last name K..., a ZIP code in Massachusetts
09:00:22 [info] [LLMGateway] -> openai/gpt-oss:20b (513 chars, json)
09:00:27 [info] [LLMGateway] <- openai/gpt-oss:20b 4565ms (in 444 / out 20 tokens)
got:   {"firstName":"Piper","lastName":"Kavanaugh","postalCode":"02115"}
asked: first name A..., last name I..., a ZIP code in Vermont
09:00:27 [info] [LLMGateway] -> openai/gpt-oss:20b (507 chars, json)
09:00:34 [info] [LLMGateway] <- openai/gpt-oss:20b 7196ms (in 623 / out 22 tokens)
got:   {"firstName":"Arielle","lastName":"Iversen","postalCode":"05332"}
asked: first name C..., last name C..., a ZIP code in Montana
09:00:34 [info] [LLMGateway] -> openai/gpt-oss:20b (507 chars, json)
09:00:40 [info] [LLMGateway] <- openai/gpt-oss:20b 6082ms (in 506 / out 20 tokens)
got:   {"firstName":"Cassandra","lastName":"Cooper","postalCode":"59014"}
  ✓  1 [chromium] › src/tests/aiTests/checkout-ai.local.spec.ts:16:5 › three AI customers from a local model (17.8s)

  1 passed (18.4s)
Call The prompt asked for The model gave Initials and five digits
1 P, K, a ZIP code in Massachusetts Piper Kavanaugh, 02115 yes
2 A, I, a ZIP code in Vermont Arielle Iversen, 05332 yes
3 C, C, a ZIP code in Montana Cassandra Cooper, 59014 yes

Each prompt asked for other initials and another state. Each customer's initials matched the prompt, and each ZIP code had five digits; whether a ZIP code belongs to its state was not checked.

Faker or AI? Asked what the benefit of generating via AI is over the Faker library, the instructor pointed back to the data just shown: can you generate this kind of data with FakerJS? To be honest, it is very difficult, or even not possible, to generate very specific data for your company.

09

Tasks and announcements

  • Homework: a TTA bank test. TTA bank is a project website the instructor is giving to everyone. You sign in (the login details are in the task post) to a balance of 50,000 rupees, and send money from Send money; the dashboard balance should then be less. Write an end-to-end test for it: in the end-to-end tests, create a folder named TTA bank, and add the test there. The important part: in Send money, you can generate the note with AI. Check both entries, the amount debited and the amount credited, as the instructor answered a learner. Then add the task to your repository.
  • The TA will share the task today, with a video or a voice explanation if possible.
  • Making a key is very simple, the instructor said: sign in on the provider's site and create one on its API keys page. The .env.example block above names the pages for Groq, OpenRouter and DeepSeek.
  • Watch the three recordings this weekend: Playwright MCP, Playwright's AI agents and Playwright CLI. The TA was asked to pin them in the group.
  • Next class: agents after the Playwright run. An RCA agent, which can do the RCA for you, and a flaky test case finder, both added to this framework. The class will also create a self-healer AI agent, which will try to heal a test whenever it fails, and rerun it. The deck's 12 November section ends with the same three.
  • Jenkins and GitHub Actions: sessions on running this framework in CI/CD, on Jenkins and on GitHub Actions, start next week, the instructor thinks. Everyone was asked to attend them.
  • Extra sessions: there might be some certification sessions too, this week or next. The timings will be shared.
  • New on the platform for this batch: a public chat for Playwright 3x in Tester Town, QA battles, the Bug Bash cafe, interview questions and answers by years of experience, and a resume builder.
  • The code: LLMGateway.ts, the prompt and the aiTests folder in the batch repo.

Task. The TTA bank homework: an end-to-end test that sends money with a note generated by AI, checks both entries, and goes into your repository.

Show the answer

Answer. Build it from the class's own pieces. In the end-to-end tests, make a folder named TTA bank and write the test there. Sign in with the details from the task post, and send money from Send money. Ask the LLM for the transfer's note through the class's gateway, the way checkoutCustomerAI() asks for a customer: a prompt, then ask() for plain text or askJson() for JSON, then a check on what comes back. Then check both entries, as the instructor asked: the amount debited and the amount credited, and the balance on the dashboard, which should be less. Push the folder to your repository.

What is the benefit of generating test data via AI over the Faker library?

Worth practising too: When would you pick an open-source model, and when a closed-source one? What makes an AI agent more than a single call to an LLM? Where can an AI agent sit around a Playwright run?

Show the answer

Answer. Faker always creates fake data that only looks similar. AI can generate data specific to your company and close to real data, such as a car's VIN, chassis and engine numbers, or a bank account or IBAN number, which is very difficult, or even not possible, with Faker. You describe the data in a prompt and ask for JSON only. Pick an open-source model for a task in bulk, where cost matters, such as the OCR for 3,000 test cases that verify a PDF's text, and a closed-source one when the task must be specific or accurate, such as code generation. An AI agent has a brain, memory and tools; the class's data generator is called an agent, but it is not fully one yet: it becomes one when it uses memory and tools. Agents can sit before a run, to generate the test data, or after it, to work on the results: an RCA agent, a flaky test finder, a self-healer.

Will it generate a credit card or IBAN number, the kind net banking uses?

Worth practising too: Your company has approved only Groq and OpenRouter: how does the class's gateway enforce that? A model wraps its JSON in a fence and adds a sentence: what does askJson() do? Why does checkoutCustomerAI() fill placeholders before every call? You have no API key: how can you still try the gateway?

Show the answer

Answer. Absolutely yes, the instructor said, and asked Ollama for one in JSON only. For the allowlist, set LLM_ALLOWED_PROVIDERS=groq,openrouter: the gateway then refuses any other provider, even one named in LLM_PROVIDER, with an error such as Provider "deepseek" is not in LLM_ALLOWED_PROVIDERS (groq, openrouter). For JSON, askJson() asks for JSON only, and extractJson() takes the JSON out of a fence or from between the chatter; if there is no JSON at all, askJson() throws LLMGatewayError with the raw reply attached. The placeholders get random initials and a state from Faker, because the same prompt gave the same customer every time. Without a key, the gateway spec skips its LLM test; to try it anyway, point the gateway at a local model with its baseUrl option, as these notes did with Ollama.