Playwright AI agents - LLM-driven authoring and execution

1What is a Playwright AI agent

An LLM-driven runner that turns natural-language goals into Playwright actions and verifies the result. Think test author + test runner glued together by a small reasoning loop.

The agent runs roughly four phases: perceive (snapshot DOM, accessibility tree, or screenshot), plan (LLM proposes the next Playwright action - click, fill, expect), act (we call the actual Playwright API), and observe (did the action succeed, did the page advance, is the assertion green). A failed observation feeds back into the next plan call so the loop self-corrects.

The two design choices that change everything: (a) what you feed the LLM as page context - full DOM, ARIA tree, screenshot, or all three - and (b) how strictly you constrain the action it can return - free text vs JSON tool calls vs typed Playwright commands. Strict tools + ARIA tree is the cheap, fast, boring default. Free-form vision is the slow, expensive, brilliant backup.

2Stagehand-style API on top of Playwright

A thin wrapper that adds ai.act, ai.observe, and ai.extract to your Playwright Page. You keep all the existing page.goto, expect, page.locator APIs and only reach for AI when the locator is fragile or unknown.

The mental model: stop the agent from generating code, let it generate resolved locators at runtime. The first time the agent sees "the green checkout button" it asks the LLM, gets back a CSS / ARIA locator, runs it, caches it. Next run it uses the cached locator and skips the LLM entirely. Cost amortises to near-zero after the first green test run.

// Illustrative TTA snippet, not a copy from upstream.
await page.goto('https://app.thetestingacademy.com/playwright/ttacart-ai/');
await ai.act('add the highlighted SKU to the cart');
await ai.act('open the cart drawer');
const total = await ai.extract({ schema: { total: 'number' } });
expect(total.total).toBeGreaterThan(0);

3browser-use and web-agent libraries

A different shape - instead of decorating Playwright with AI calls, you hand the entire browser to the agent and let it drive on its own. The agent generates a multi-step plan, executes step by step, re-plans on error.

The agent gets one goal ("book the cheapest TTAStays room for next weekend") and gets to choose every Playwright action - including click, fill, scroll, extract, and finish. The library exposes Playwright primitives as tools, and the LLM decides the sequence. Where Stagehand is a hammer, browser-use is a robot arm.

Cost goes up fast - every step is a fresh LLM call - so production usage caps max_steps (typically 10-30), runs a cheap small model first, and only escalates to a large model on retry. Some libraries cache the final action plan and let you replay it deterministically the next run - you got the AI to "discover" the test, and now you can re-run it like a normal Playwright script.

4Self-healing locators

The single most-loved AI feature in Playwright land. When a locator fails, the AI proposes 3 fallback locators using the current DOM, ranked by stability. You retry with the top pick and log the heal so a human can promote it later.

The flow lives one layer below your test: the locator wrapper catches not-found / not-visible, asks an AI client for 3 candidates, applies the top one, and only throws if all three fail. The healed locator is written to a log file but the source spec is left untouched - that way a heal never silently masks a real regression. A weekly maintainer PR promotes the most-used heals into the codebase as real selectors.

Detailed write-up of how we wire this in V2 lives in Framework + AI #heal, plus the Top 5 AI features card with a sequence diagram and an excerpt from UtilElementLocator.ts.

// On primary locator fail, ask AI for 3 candidates against the current DOM.
const dom = await page.locator('main').innerHTML();
const candidates = await ai.heal({ original: '#sku-1842', intent: 'click', dom });
await page.locator(candidates[0]).click();    // try the best one
log.heal({ original: '#sku-1842', healed: candidates[0] });

5Prompt to spec - natural language to .spec.ts

You write the test in English. The agent reads the target page (or a recorded session), generates a real Playwright spec file, runs it, captures the result. Spec files live in source control - no AI at test time.

This is the design pattern most teams should default to: AI helps you write the test, but the test itself is plain Playwright code that runs without any LLM dependency in CI. You get the speed of AI authoring with the determinism of vanilla Playwright at runtime. The same generator can re-emit specs when the page changes - you keep the English prompt as source of truth, treat generated .spec.ts as build output.

How we use this in the TTA framework + AI build is described in Framework + AI #features (prompt-to-spec card) with a sequence diagram. The TTACart-AI sandbox includes a live generator demo.

6Visual reasoning - the vision LLM as a tester

Instead of feeding the LLM DOM text, you feed it a screenshot. The model can verify layout, spot z-index bugs, catch dark-mode contrast issues, and even reason about which button to click when the DOM is opaque (canvases, custom drawn UI).

Vision is the right tool when the DOM lies - canvas-rendered charts, custom GL renderers, PDF viewers, drawn-on-canvas tables. It's also useful for layout assertions ("the price label must be within 20px below the SKU title"). The trade-off: a vision call is 5-20x the token cost of a DOM call, so most teams cap it to once per spec, used only when DOM-based reasoning fails.

A standard recipe: page.screenshot() -> send to vision LLM with a tight prompt ("return the bounding box of the green checkout button in pixels") -> convert pixels back to Playwright page.mouse.click(x, y). Brittle? Yes. Catches bugs DOM-only tests miss? Also yes.

const png = await page.screenshot({ fullPage: false });
const box = await ai.visionLocate({
  image: png,
  intent: 'the green Checkout button at the bottom of the cart',
});
await page.mouse.click(box.cx, box.cy);
await expect(page).toHaveURL(/\/payment/);

7Cost + privacy guards

Five guards that stop AI agents from leaking customer data or burning the AWS bill. Every one of them lives in your AIClient wrapper, not in the upstream provider library.

1. Proxy every cloud call. Route requests through a Cloudflare Pages function or equivalent. Keys never touch the test machine. We do this already for our chat demo via functions/api/chat.js.
2. Redact before send. Strip emails, phone numbers, postal codes, card last-4s from DOM snapshots and trace summaries. Local providers (Ollama, LM Studio) skip the redactor - data never leaves the box anyway.
3. Hard token budget. A per-run cap on input + output tokens. Hit the cap, fail fast. No surprise $500 bills.
4. Cache responses. Hash {model, system, prompt}, store the result, return it on cache hit. Same query in CI never pays twice.
5. Audit every prompt + response. Append-only log so a security review can answer "what did we ask the model about prod data last Tuesday".

Full implementation in Framework + AI #guards. The principle: AI agents are not exempt from the same data handling rules as the rest of your test suite. They get extra scrutiny, not less.

8When NOT to use AI agents

AI is not free. For stable selectors, deterministic flows, and well-defined assertions, vanilla Playwright is faster, cheaper, and easier to debug. Use this checklist to decide.

1. You already have stable test-ids. A data-testid attribute on every interactive element makes AI redundant. Use the test-id.
2. You need byte-identical reruns. Even temperature=0 LLM calls have provider drift. If you cannot accept any non-determinism, generate the spec once and check it in.
3. Runtime budget is under 30s per spec. First-run AI calls add 2-10s each. Cache helps, but cold starts will blow the budget.
4. You can't ship traffic to a 3rd party. Healthcare, finance, defense. Run a local model or skip AI entirely.
5. Your DOM is small and stable. A 50-element TTAStays booking form rarely needs healing. Save the AI for the 5,000-element data table.

Good heuristic: use AI to generate tests, use vanilla Playwright to run them. The generated spec is yours to read, edit, commit, and replay forever.

Provider switcher - 7 models behind one interface

Same as the multi-model adapter in the Framework + AI doc. Switch by setting process.env.TTA_AI_MODEL. Local providers (Ollama, LM Studio) need no key - data never leaves the machine.

Full table with endpoints + auth in Framework + AI #adapter.

Diagram - the agent loop

One mermaid diagram. Perceive, plan, act, observe, decide-to-continue. Vanilla Playwright runs the action; the LLM only owns the plan step.

flowchart LR
  G[Goal in English] --> P[Perceive: DOM / ARIA / screenshot]
  P --> LP[LLM: propose next action]
  LP --> A[Playwright: click / fill / scroll / extract]
  A --> O[Observe: assertion or page advance]
  O -- goal done --> Z[finish + emit trace]
  O -- not yet, retry --> LP
  O -- failed, replan --> RP[LLM: rewrite plan with new context]
  RP --> A
  O -- budget or max_steps --> X[fail with last screenshot + log]

  classDef pw fill:#d1fae5,stroke:#16a34a,color:#111
  classDef ai fill:#ede9fe,stroke:#8b5cf6,color:#111
  classDef end1 fill:#fef9c3,stroke:#f59e0b,color:#111
  class P,A,O pw
  class LP,RP ai
  class Z,X end1