AI Agents for QA: build a testing agent that actually acts

Watch the agent loop turn a goal into action

Pick a flow and press Play. Each step lights up one part of the reason-act-observe loop, so you can see how an outcome you asked for becomes a chain of real tool calls.

The roadmap

Agency is a dial, not a switch. Turn it up one notch at a time: assist, then act, then heal, then orchestrate, keeping a human in the loop until each stage earns trust.

1. What an AI agent is, for QA

Strip away the marketing and an AI agent is three things wired together: a language model, a set of tools it is allowed to call, and a loop that runs until the job is done. That is the whole idea. The model does not test your app by itself. It decides, one step at a time, which tool to call next, and something outside the model actually opens the browser, runs the suite, or reads the log. ai agents for qa are just this pattern pointed at testing work: the same models you already prompt, given hands and a feedback loop.

The distinction that matters is agent versus prompt, and it is not a matter of degree. A prompt is one shot. You hand the model a spec, it hands you back text, and the exchange is over. It never finds out whether the test it wrote actually runs. An agent gets a goal and a toolbox, takes an action, reads the real result of that action, and uses the result to choose what to do next. It can be wrong on step one and recover on step three, because it is reacting to what actually happened instead of guessing from the prompt alone. That recovery is the entire reason to reach for an agent.

The loop is the whole trick

Function calling is what lets a model act. You describe each tool to the model as a named function with typed arguments (a JSON schema), and instead of replying in prose the model can emit a structured request: call browser_click with these arguments. The model itself executes nothing. Your runtime, usually called the harness or orchestrator, runs the call, captures the result, and appends that result back into the conversation as an observation. Then the model runs again with that new information and picks the next step. Reason, act, observe, repeat. That cycle is what separates ai testing agents from a chat window that can only describe what it would do.

Written as a loop it is almost boring, which is exactly the point:

while not done and steps < MAX_STEPS:
    step = llm.next(history, tools)         # model decides one action
    if step.kind == "tool_call":
        result = run_tool(step.name, step.args)  # harness executes it
        history.append(observation(result))    # feed the truth back in
    else:
        done = True                          # model signals the goal is met
    steps += 1

Three details in that loop carry all the weight. The tools you pass in define the outer edge of what the agent can do: no browser tool, no browsing, no matter how badly it wants to. The observation you append has to be honest, because if you hand back a truncated or misleading result the model reasons from a lie. And MAX_STEPS plus a budget cap are not optional decoration. An agent with no stop condition will loop happily forever, and every loop is a metered model call. This is the first thing that surprises teams moving from prompts to agentic testing: you are now paying per step and per token, so a runaway loop burns real money instead of just wasting a reply.

One shot versus a loop

The same underlying model behaves very differently in the two modes. A one-shot prompt is fast, cheap, and stateless, and it is genuinely good at drafting from a spec. An agent loop is slower and pricier, but it can cope with reality shifting under it while it works.

Notice that the failure modes differ, which changes how you supervise each. A one-shot prompt fails by being confidently wrong: it invents a selector that looks plausible and you only find out in CI. Autonomous test agents fail the opposite way, by thrashing between two fixes, declaring victory before the suite is green, or spending twenty steps on a problem a human would abandon. Both need a human in the loop. They just need different kinds of attention.

What the loop looks like against a real app

Concreteness helps here. Point an agent at the practice pages at app.thetestingacademy.com/playwright with a browser tool (a Playwright MCP server exposes exactly these actions), a file writer, and a shell, then give it one goal: write a passing test for the login form. A real trace looks like this:

# goal: write a passing test for the login form at /playwright
> browser_navigate  url="app.thetestingacademy.com/playwright/login"
  obs: page loaded; form has #username, #password, button[type=submit]
> browser_snapshot
  obs: textbox "Username" ref=e12; textbox "Password" ref=e15; button "Log in"
> write_file  tests/login.spec.ts
  obs: wrote 24 lines
> run_shell  "npx playwright test login.spec.ts"
  obs: 1 failed - locator "#user" resolved to 0 elements
> edit_file  tests/login.spec.ts   # #user -> #username
> run_shell  "npx playwright test login.spec.ts"
  obs: 1 passed (2.3s)
> done

Read that trace as a QA engineer and you will recognize your own inner loop: open the page, look at the actual DOM, write the test, run it, read the failure, fix the selector, run again. The agent did not know #username was the correct id until it navigated and took a snapshot. That is the whole advantage of the loop over a prompt: it grounds every step in the real application instead of the model's memory of how login forms usually look. This is why ai testing agents are far more dependable at authoring against a live app than a one-shot generator that never opened the page and simply pattern-matched a guess.

Where agents earn their keep, and where they do not

Agents are strong at mechanical, bounded, verifiable work: the parts of QA that are tedious rather than genuinely hard. They are weak substitutes for judgment: the parts that decide what "good" even means. Keep this line bright, because most bad agent deployments come from crossing it.

The pattern running through that table is verifiability. Anything an agent can check against ground truth (does the test run, does the selector resolve to one element, does the page load) is fair game for autonomous test agents, because the loop can confirm its own work. Anything that requires knowing your product, your users, and your tolerance for risk stays with you. An agent can cluster fifty red tests into three root causes in a minute, which is real leverage on triage. It cannot tell you that the checkout regression matters more than the tooltip one, because "matters" is a business fact, not a value the test runner returns.

Hold both truths at once. ai agents for qa are not a smarter autocomplete, and they are not a replacement for testers. They are a way to spend model calls turning your intent (what to test, what "correct" means) into executed, observed, self-corrected work. The rest of this guide is built on exactly this loop: give the agent the right tools, honest ground truth to observe, and hard stop conditions, then keep the judgment calls for yourself.

2. The agentic loop, and the agent architecture

An agent is not a smarter prompt. It is a control loop with a language model wired into one step of it. Strip away the branding and the ai agents for QA you build or buy all run the same four-beat cycle: perceive the current state of the system under test, reason about what to do next, act on it, then perceive again. The model supplies judgment at exactly one beat of that cycle. The loop, the tools, and the runtime around it supply everything that turns that judgment into repeatable work.

Get this picture right early and you avoid the most expensive mistake in agentic testing: treating the model as the product and the scaffolding as plumbing. It is the reverse. The base model is a commodity you rent by the token. The loop and the tool set are what you engineer, and they are where reliability is won or lost.

Perceive, reason, act, repeat

Take a concrete goal: "sign in on the practice site and confirm the dashboard renders." Point the agent at app.thetestingacademy.com/playwright and watch one turn of the cycle.

Perceive. The runtime hands the model an observation of the current page. For a browser agent this is usually not a screenshot. It is a structured accessibility snapshot: a compact tree of roles, accessible names, and reference ids, scoped to that snapshot, that the tools can act on directly.

# browser_snapshot returns an accessibility tree with per-snapshot refs
- textbox "Username" [ref=e5]
- textbox "Password" [ref=e6]
- button "Log in" [ref=e7]
- alert "Invalid credentials" [ref=e9]   # present only after a failed attempt

Reason. The model reads that snapshot together with the goal and the history so far, and emits a single decision expressed as a typed tool call: type into the username textbox at ref e5, type into the password textbox at e6, click the button at e7. It is not touching the browser. It is proposing the next action in a structured form the runtime can execute.

// the model's chosen action, as typed tool calls
browser_type({ element: "Username field", ref: "e5", text: "[email protected]" })
browser_type({ element: "Password field", ref: "e6", text: "..." })
browser_click({ element: "Log in button", ref: "e7" })

Act. The runtime executes each proposed call against the real browser and captures what actually happened: a new snapshot, a console error, a network response, a thrown timeout. That captured result is ground truth, not the model's expectation of it.

Repeat. The fresh result becomes the next perception, and the cycle turns again. It keeps turning until the model stops asking for tools and returns a final answer (goal reached), or until a guard trips: a step ceiling, a token budget, a wall-clock timeout, or the same error twice in a row. That stopping logic lives in your code, never in the model's good intentions.

const MAX_STEPS = 40;
let messages = [ userGoal ];

for (let step = 0; step < MAX_STEPS; step++) {
  // reason: one model turn, with the tool schemas advertised
  const reply = await model.create({ messages, tools });
  messages.push({ role: "assistant", content: reply.content });

  // stop: the model returned prose, not a tool call -> goal reached
  const calls = reply.content.filter(isToolUse);
  if (calls.length === 0) break;

  // act: run each requested tool, collect the results
  const results = [];
  for (const call of calls) {
    const output = await runtime.dispatch(call.name, call.input);
    results.push({ type: "tool_result", tool_use_id: call.id, content: output });
  }

  // perceive: feed the results back as the next observation
  messages.push({ role: "user", content: results });
}

Notice what the model does and does not do. It never clicks anything. It reads structured text and writes structured text. Every actual effect on the system under test passes through a tool the runtime controls. This separation is the whole game for autonomous test agents: the stochastic part proposes, the deterministic part disposes, and the observation after each act keeps the two honest with each other.

What a perception actually is

The quality of the perceive step sets the ceiling on everything downstream. Feed the model a raw screenshot and it has to localize a login button by pixels, which drifts the moment a font renders differently or a cookie banner pushes the layout down. Feed it an accessibility snapshot and it selects a button by role and name, which is stable across renders and maps cleanly to a real locator. This is why serious ai testing agents lean on the accessibility tree, the DOM, console logs, and network traces as their primary senses, and treat screenshots as a fallback for the genuinely visual. The perception format is not a detail. It is a reliability lever you pull in the tool layer, before the model reasons at all.

The four components

Every agent, from a fifty-line script to a commercial platform, is assembled from four parts. Naming them makes it obvious which part to fix when a run goes wrong.

The runtime deserves emphasis because it is the part people forget. It is the difference between a slick demo and a system you can point at two thousand test cases overnight. Tools are commonly delivered over MCP, the Model Context Protocol: a small JSON-RPC standard where a server advertises a set of typed tools and any compliant runtime can connect and call them. The Playwright MCP server, for instance, exposes the browser verbs an agent needs, and those verbs map neatly onto the loop beats.

Why reliability lives in the loop, not the model

Here is the claim to internalize: a bigger model does not fix a flaky autonomous test agent. A tighter loop and better tools do. The reason is what a language model fundamentally is, a next-token predictor with no direct access to truth. Ask a bare model "did the login succeed" and it will produce a fluent, confident, and entirely ungrounded guess. Wrap that same model in the loop and the tool result answers the question for it: the snapshot either shows the dashboard or it shows the "Invalid credentials" alert at ref e9. The tool call is the reality check that keeps the reasoning honest, turn after turn.

So the engineering job is to spend your determinism budget wisely. Push every scrap of certainty into the tools, where a click is exactly a click and an assertion is exact, and leave the model only the genuinely ambiguous work of perception and planning. In practice that means:

• Constrain the tool surface to what the task actually needs, so the model cannot wander into destructive calls.
• Validate tool inputs and make tools idempotent, so a retried click cannot corrupt state.
• Add an explicit verify beat that asserts on the world, instead of trusting the model's self-report.
• Cap the loop, so a confused agent burns ten steps, not ten thousand.

This also explains the shape of the market. Most ai testing agents on sale today share a handful of the same base models; what differs is the harness around them, the perception format, the tool set, the stop conditions, and the verification. The harness is the moat, and it is where their reliability comes from too. So when you evaluate or build ai agents for QA, judge the loop and the tools first and the model second. The model is the easy part to swap. The scaffolding is the part that decides whether an agent can be trusted with a real suite, and it is the rest of this guide's subject.

3. The AI-agent-for-QA maturity roadmap

Most teams do not stall on agentic testing because the models are weak. They stall because they jump from a tool that only suggests to a system that acts, with nothing built in between: no trust earned, no guardrail in place. Treat adoption of ai agents for qa as a ladder of autonomy, not a feature checklist. Each rung buys one unit of independence and, in exchange, hands you one specific failure mode you now have to contain. The rule that keeps you safe is boring on purpose: promote to the next stage only after the current one has stopped surprising you.

The five stages below run from an assistant that cannot touch anything on its own to a scheduled fleet that works while you sleep. For each stage, hold three questions in view: what new capability it buys, what risk shows up when it goes wrong, and the single guardrail that makes that risk acceptable. Here is the whole ladder before we walk it.

Stage 1: Assistant (it suggests, you run)

This is the in-IDE tier: GitHub Copilot, Cursor, Claude Code. It autocompletes a locator, drafts an assertion, or explains a flaky stack trace. It is the honest entry point to ai agents for qa precisely because it changes nothing on its own. You are still the compiler, the runner, and the reviewer. The capability is speed of authoring, not autonomy.

The risk is confidently wrong output: a hallucinated getByRole name, a Playwright flag that never existed, an assertion that reads plausibly but checks nothing. The failure here is as much social as technical, because the temptation is paste-without-reading. The guardrail is to keep the loop read-and-suggest. Nothing executes unattended, every suggestion lands as a diff a human accepts, and you treat the assistant like a strong junior whose pull requests you always read line by line.

Stage 2: Generator (one grounded test, under review)

Now the agent writes a whole test, not a single line. The word that carries all the weight is grounded. A grounded generator reads the real target before it writes: the live DOM or the accessibility tree, not its training memory. Point it at the login form on app.thetestingacademy.com/playwright, let it pull the page snapshot through the Playwright MCP server, and it emits role-based locators such as getByLabel('Email') and getByRole('button', { name: 'Sign in' }) that match what is actually rendered rather than what the model guessed.

The risk is over-generation and hollow checks. Ungoverned ai testing agents love to produce fifty specs nobody will maintain, or a test that navigates, screenshots, and asserts nothing that matters. Snapshot tests baked against implementation detail rot by the next sprint. The guardrail: one test per pull request, grounded in the real page, under human review, with a required assertion that goes red when the behavior actually breaks. If you cannot break the feature and watch the test fail, the test is theater.

Stage 3: Healer (fixes selector drift, stops on real bugs)

The capability here is maintenance. When a class or an id changes but behavior is intact, the healer proposes a fresh locator and reruns, so a cosmetic refactor does not turn the whole suite red overnight. Selenium teams know this pattern from Healenium; the modern version leans on role and label locators that survive markup churn far better than brittle XPath.

The risk is the defining trap of self-healing. If the agent is allowed to touch assertions or expected values, it will "fix" a broken checkout by loosening the check, and a genuine regression sails through green. A healer that can edit assertions is not a healer, it is a bug concealer. The guardrail: heal the locator layer only, never assertions, expected text, or state. On an assertion failure or an unexpected page state, stop and report; do not heal. Cap the heal attempts, and make every heal a reviewable diff instead of a silent rewrite.

// Healer discipline: repair the locator layer, never the assertion.
async function resolveOrHeal(page, intent) {
  const primary = page.getByRole('button', { name: 'Sign in' });
  if ((await primary.count()) > 0) return primary;

  // Locator drifted. Ask the agent for a new locator for the SAME intent,
  // grounded in the current accessibility tree, with a hard attempt cap.
  const healed = await agentSuggestLocator(intent, {
    snapshot: await page.accessibility.snapshot(),
    maxTries: 2,
  });
  recordHealDiff(intent, healed);   // reviewable, not silent
  return healed;
}

// The assertion stays hand-written and untouchable by the healer.
await expect(page.getByText('Welcome back')).toBeVisible();

Stage 4: Multi-agent (planner + generator + healer)

Here roles split across cooperating agents. A planner decomposes a feature into scenarios, a generator writes each one, a healer maintains them, and a critic reviews before anything merges. This is the point where agentic testing starts to feel like a small team rather than a single tool, and where orchestration, not raw model quality, decides whether you get value.

The risk is compounding. A weak plan fans out into forty weak tests. Errors chain from one agent to the next, token spend and dollar cost balloon, and non-determinism makes a red run hard to reproduce. Without traceability you cannot say which agent made which decision. The guardrail: bound the fan-out with a hard cap on sub-agents per run, gate every merge behind a critic or a human, cap the per-run budget, and log the full trace so each decision is attributable. A human approves the plan before the swarm ever acts on it.

Stage 5: Scheduled and autonomous, under guardrails

Now the suite runs on a schedule or on every deploy. Autonomous test agents heal drift, triage failures, file a structured bug, open a pull request with the fix or the new test, and then ping a human. This is the payoff of the entire ladder: coverage that largely maintains itself, watching a target like the practice flows on app.thetestingacademy.com/playwright without someone babysitting the run.

The risk is unattended action at scale. One misfire files a hundred false bugs, auto-merges a bad heal, or drains an API budget overnight. Alert fatigue sets in and people begin ignoring the very agents meant to protect them. The guardrail is layered: autonomous test agents write to a pull request or an issue, never to main. Enforce hard budget caps per run and per day, a kill switch behind a feature flag, a confidence threshold that escalates uncertain cases to a human, and a bounded blast radius (read-only against production, write only to a sandbox). An audit log makes every autonomous action reviewable and reversible after the fact.

You do not have to reach stage 5 to win. Most QA teams get durable, defensible value at stage 2 or stage 3, where a grounded generator and a locator-only healer quietly cut authoring and maintenance cost without ever acting alone. Advance a rung only when the current one has gone quiet: when the generator's tests clear review without edits for a fortnight, when the healer has not hidden a single bug. And because regressions in behavior and in trust both happen, keep the ability to demote a stage as a first-class control, not an afterthought.

4. Agent frameworks and stacks for QA

"Agents for QA" collapses three layers that are worth separating before you pick a tool. At the bottom is a transport: the Model Context Protocol (MCP), the open standard that lets an agent call tools like a browser, a filesystem, or a database over one common wire. In the middle sits orchestration: code frameworks (CrewAI, LangGraph) that decide which model does what, in what order, with what memory. On top is a visual layer (n8n, LangFlow) that exposes the same orchestration as a canvas non-engineers can read. Most teams do not need all three on day one. You reach for each based on how much control, durability, and cross-team visibility the job actually demands, so the survey below is ordered from the lightest touch to the most industrial.

MCP coding agents driving Playwright MCP

An MCP-based coding agent is the fastest way to get hands-on with agentic testing. Claude Code and Cursor already live in your terminal or editor; point them at the Playwright MCP server (Microsoft's @playwright/mcp) and they gain a real browser. The server is accessibility-tree first: instead of feeding screenshots to the model, it returns a structured snapshot of the page (roles, names, refs), then exposes tools like browser_navigate, browser_click, browser_type, browser_snapshot, and browser_wait_for. The agent reasons over that tree the way a screen reader would, which keeps its clicks deterministic and the selectors it generates stable. Wiring it up is one line in Claude Code or a few lines of JSON anywhere else:

# Claude Code: register the Playwright MCP server
claude mcp add playwright -- npx @playwright/mcp@latest

# Cursor / any MCP client: .mcp.json
{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest"]
    }
  }
}

Now you can say: "open app.thetestingacademy.com/playwright, complete the login widget, and write me a @playwright/test spec that asserts the success toast." The agent drives the live practice page, observes the accessibility tree, and emits runnable test code. This is where most engineers should start with ai agents for qa: exploratory passes over a new build, reproducing a customer bug from a plain description, or converting a manual test case into a spec. Reach for it when the work is interactive and individual. It is deliberately not a durable pipeline; close the session and the run state is gone. (Our Playwright MCP for QA deep-dive covers headless CI runs, storage-state auth, and trace capture.)

CrewAI: role-based crews

CrewAI models a workflow as a team of role-playing agents. You declare an Agent with a role, goal, and backstory, give it tools and a model, bind Task objects to agents, then group them into a Crew that runs sequentially or hierarchically (a manager agent delegating to workers). That mental model maps cleanly onto a real QA org, which is why it is a comfortable first framework for building ai testing agents. The triage / plan / test crew is the canonical shape: a Triager classifies an incoming defect, a Planner turns it into a regression plan, and an Author writes the Playwright steps.

from crewai import Agent, Task, Crew, Process

triager = Agent(
    role="Bug Triager",
    goal="Classify each report by severity (S1-S4) and component",
    backstory="A senior SDET who has triaged thousands of defects.",
)
author = Agent(
    role="Test Author",
    goal="Turn a triaged bug into a runnable Playwright spec",
    backstory="You write minimal, high-signal regression tests.",
)

triage = Task(description="Report: {report}. Assign severity and component.",
              expected_output="JSON with 'severity' and 'component'.", agent=triager)
write  = Task(description="Using the triage, produce a @playwright/test spec.",
              expected_output="One .spec.ts file, ready to run.", agent=author)

crew = Crew(agents=[triager, author], tasks=[triage, write],
            process=Process.sequential)
crew.kickoff(inputs={"report": open("bug.md").read()})

Reach for CrewAI when the work decomposes into distinct roles and you want a declarative, readable structure with almost no plumbing. Because each agent carries its own goal and backstory, the prompts stay legible and the division of labor is explicit in the source rather than hidden in one mega-prompt. The trade-off is control: crews give you sequential or manager-delegated flow, not arbitrary branching or tight retry loops, and a chatty crew can spend real tokens agreeing with itself. When your QA workflow is mostly "these roles, in this order," CrewAI is the least ceremony for the most structure. (The CrewAI for test automation walkthrough wires the tools and a full triage crew.)

LangGraph: stateful multi-step agents

LangGraph, from the LangChain team, is what you graduate to when linear crews stop expressing your control flow. It models an agent as a state machine: a typed State object, nodes that read state and return partial updates, and edges (including conditional edges) that decide what runs next. Crucially it supports cycles, so an agent can loop, and checkpointing, so a long run can persist, pause for approval, and resume. That combination is exactly what autonomous test agents need. A self-healing loop is the natural example: generate a spec, run it, inspect the failure, patch the spec, run again, and stop when it goes green or a retry budget is hit.

from langgraph.graph import StateGraph, START, END
from typing import TypedDict, Annotated
import operator

class TestState(TypedDict):
    spec: str
    attempts: int
    passed: bool
    logs: Annotated[list, operator.add]   # reducer: append, don't overwrite

def run_test(state):                       # node: execute the current spec
    result = execute(state["spec"])
    return {"passed": result.ok,
            "attempts": state["attempts"] + 1,
            "logs": [result.output]}

def route(state):                          # conditional edge: loop or stop
    if state["passed"] or state["attempts"] >= 3:
        return END
    return "fix"

g = StateGraph(TestState)
g.add_node("run", run_test)
g.add_node("fix", fix_test)                   # patch spec from logs[-1]
g.add_edge(START, "run")
g.add_conditional_edges("run", route, {"fix": "fix", END: END})
g.add_edge("fix", "run")                    # cycle: fix feeds back into run

app = g.compile()
app.invoke({"spec": spec, "attempts": 0, "passed": False, "logs": []})

The conditional edge is the whole point: control flow lives in code you can read and unit-test, not buried in a prompt where you cannot assert on it. Reach for LangGraph when your agentic testing needs real branches (a failure routes to a debugger node, a pass routes to a reporter), bounded retries, durable state across a run that might take minutes, or a human gate before anything destructive runs. The cost is that you write more, and at a lower level, than CrewAI asks for. (The LangGraph testing agents guide builds this self-healing loop with checkpoints and an approval interrupt.)

n8n and LangFlow: visual flows

The visual layer trades some code-level control for legibility and reach. n8n is a node-based automation canvas (self-hostable, fair-code) whose AI Agent node (LangChain-backed) can hold tools and a model, wired between triggers (webhook, schedule) and integrations (Jira, Slack, GitHub). It shines as glue: a CI job fails, a webhook fires, an agent node reads the logs and decides whether this is a genuine regression or a flake, opens a Jira issue, and posts a summary to Slack, on a schedule the whole team can watch. LangFlow is the other visual option, a drag-and-drop builder for LangChain-style flows that compiles to a runnable API, good for prototyping a triage agent or a retrieval helper over your test documentation before committing it to code.

Reach for the visual layer when the audience is cross-functional, when the value is integrating existing systems more than novel reasoning, or when a scheduled, always-on job matters more than fine-grained branching. The caution is that visual flows grow unwieldy as branches multiply, and versioning or testing the flow itself is harder than testing a Python graph. In practice, mature teams end up layered: an MCP coding agent for authoring, a LangGraph loop for the hard iteration, and n8n out front so ops can see it run. (The n8n testing workflows page has the CI-failure-to-Jira recipe end to end.)

5. Building a test-generation agent

A test-generation agent is the highest-leverage thing you can build in an afternoon. The premise is deliberately narrow: hand it a user story or a bare URL, let it drive a real browser through Playwright MCP, and get back one Playwright test that already ran green before you laid eyes on it. This is where ai agents for qa stop being a conference demo and start paying rent. We are not asking a model to imagine a page from its training data and guess at selectors. We give it eyes on the live DOM and a tight contract for what it is allowed to write. The output is not a suggestion you clean up for twenty minutes; it is a committed, passing spec you review for intent.

The whole design rests on one split: a planner that decides which single journey to cover, and a generator that turns observed page state into code. Keeping those two jobs distinct is what separates a reliable author from a model that wanders off and produces a 200-line mega-spec nobody trusts. Most flaky output from early ai testing agents traces back to skipping the plan and letting the generator improvise its way across half the site.

The shape of the agent

The agent runs a fixed five-phase loop. It plans, explores the live page, generates exactly one spec, runs it, and iterates on real failures until the test is green or a retry budget is spent. Nothing in this loop depends on the model remembering the page. Every locator it writes is earned from something it just observed. That property is what makes the difference between autonomous test agents you can leave running and a toy that needs a babysitter watching every step.

Under the hood the agent leans on a small slice of the Playwright MCP tool surface. The one tool that matters most is browser_snapshot, which returns an accessibility tree of the current page (roles, accessible names, and ephemeral ref handles) rather than a screenshot. The agent drives the page by ref during exploration, but it never writes refs into the final test, because refs are session-scoped and would be meaningless on the next run. That distinction, ref to drive versus role and name to assert, is the seam most first-time builders miss.

The agent file

Everything the agent knows lives in one instruction file. Treat it like production code: it is the spec for your spec-writer. The version below is written planner-first, with the grounded-locator rule and the one-journey guardrail stated as hard constraints rather than suggestions, because a model will quietly relax anything phrased as a preference.

# Agent: playwright-test-author

## Mission
Given a user story or a URL, produce exactly ONE Playwright test for ONE journey.

## Tools
- Playwright MCP: browser_navigate, browser_snapshot, browser_click, browser_type, browser_close
- Filesystem write: tests/*.spec.ts
- Shell: npx playwright test

## Workflow
1. PLAN. Restate the journey in <= 6 steps. If the story implies many journeys, pick the primary one and note the rest.
2. EXPLORE. Navigate to the URL. Snapshot after every state change. Never act on an element you have not seen in a snapshot.
3. GENERATE. Write one spec. Every locator maps to a role + accessible name from the latest snapshot.
4. RUN. npx playwright test .
5. ITERATE. On failure: re-snapshot, change ONE thing, re-run. Max 4 attempts, then stop and report.

## Grounded-locator rule
- Allowed: getByRole, getByLabel, getByPlaceholder, getByText, using names copied verbatim from the snapshot.
- Forbidden: CSS/XPath from memory, guessed ids, nth-child, text you did not observe.
- Cannot find it in a snapshot? Do not invent a locator. Report the gap.

## One journey per session
- One spec, one test() block, one happy path plus its success assertion.
- Do not add a second journey. End with browser_close so the next run starts clean.

## Output contract
- File: tests/.spec.ts, with a PLAN comment at the top.
- No console.log, no waitForTimeout, no test.skip.

Two choices in that file carry most of the weight. First, the retry budget is capped at four attempts: an agent that can loop forever will eventually "fix" a real bug by weakening the assertion until it passes, which is worse than a red test. Second, the file forbids waitForTimeout outright. Agentic testing tools love to paper over a race with a sleep, and a sleep is a defect you will pay for in CI a hundred times over.

The grounded-locator rule

This is the single rule that decides whether your agent is useful. A grounded locator is one whose role and accessible name were both copied from a snapshot the agent actually took. An ungrounded locator is anything the model produced from memory: a guessed CSS class, an #id it never saw, an nth-child chain, or brittle text it assumed would be on the page. Ungrounded selectors are the number-one failure mode of ai testing agents, and they fail silently: the code looks plausible, passes review, and then times out against the real site at 2 a.m.

Here is a trimmed snapshot from the sign-in form on the practice pages at app.thetestingacademy.com/playwright:

- Page URL: https://app.thetestingacademy.com/playwright/forms/practice.html
- Page Snapshot:
- main:
  - heading "Sign in" [level=1] [ref=e2]
  - textbox "Email" [ref=e4]
  - textbox "Password" [ref=e5]
  - button "Sign in" [ref=e6]
- navigation:
  - button "Sign in" [ref=e9]

The mapping is mechanical. A node reading textbox "Email" becomes getByLabel('Email'); button "Sign in" becomes getByRole('button', { name: 'Sign in' }); a placeholder-named field becomes getByPlaceholder. The agent copies the name string verbatim, it does not paraphrase "Sign in" into "Login". Because Playwright's role locators resolve against the same accessibility tree the snapshot was built from, a name that appears in the snapshot is a name the locator can find. That is the entire trick, and it is why grounding, not model size, is what makes agentic testing dependable.

Running and iterating until green

Generation is a draft, not a delivery. The agent writes the spec, then runs it with npx playwright test and reads the result the same way you would. The interesting behavior is what happens on red. A well-instructed agent does not patch code blindly; it takes a fresh snapshot, compares what it assumed against what the page reports now, and changes exactly one thing. Here is the real first run against our sign-in journey:

$ npx playwright test tests/sign-in.spec.ts

  1) [chromium] > sign-in.spec.ts:4 > registered user signs in and reaches the dashboard

  Error: locator.click: Error: strict mode violation:
  getByRole('button', { name: 'Sign in' }) resolved to 2 elements:
      1) <button type="submit">Sign in</button>
      2) <button class="nav-cta">Sign in</button>

The failure is a strict-mode violation: two buttons share the accessible name "Sign in", one in the form and one in the navigation. Grounding removed hallucination, but it cannot remove ambiguity, and this is exactly why the run-and-iterate phase exists even for a perfectly grounded draft. The agent re-snapshots, confirms both nodes, and scopes the click to the main landmark. A second correction lands on the assertion: the story said "reaches the dashboard", but the post-login snapshot showed the real heading was "Welcome back", so the agent asserts what the page says, not what the story guessed.

Four attempts is plenty for real work. If the test is still red after that, the right move for autonomous test agents is to stop and hand back a report, not to keep mutating the spec. A red test the agent could not honestly make green is a signal, sometimes about the page, sometimes about the story you fed it.

One journey per session

The last guardrail is the one people skip and regret. One agent session produces one spec, one test() block, one journey. The moment you let a session cover "sign in, then update the profile, then check the leaderboard", context bloats, snapshots pile up, locators from page three leak into assertions about page one, and you get the flaky mega-spec you were trying to avoid. Constraining scope also keeps the token bill sane and the diff reviewable in under a minute.

Enforce it in the agent file, not by hoping. End every session with browser_close so the next journey starts from a clean browser and a clean context window. When you want breadth, run the agent many times with different stories, not once with a giant one. This is the same discipline that separates well-behaved ai agents for qa from the ones that quietly rot into an untrusted suite.

The spec it produces

Here is the grounded, green output for the sign-in journey. Notice what is absent: no CSS soup, no sleeps, no refs, no second journey smuggled in.

import { test, expect } from '@playwright/test';

// PLAN: registered user signs in from the practice form -> dashboard
test('registered user signs in and reaches the dashboard', async ({ page }) => {
  await page.goto('https://app.thetestingacademy.com/playwright/forms/practice.html');

  await page.getByLabel('Email').fill('[email protected]');
  await page.getByLabel('Password').fill('Practice@123');
  await page.getByRole('main').getByRole('button', { name: 'Sign in' }).click();

  await expect(page.getByRole('heading', { name: 'Welcome back' })).toBeVisible();
});

This is a test a human would have written on a good day, and it took the agent about a minute. What still needs your eyes is the assertion. The agent proved the page reaches a "Welcome back" heading, but only you know whether that is the business-meaningful proof of success or merely the first visible change after submit. Grounding fixes locators and locates the truth on the page, it does not decide intent. Treat the emitted spec as a strong draft from a fast junior: review the assertion, confirm it proves the thing that matters, and commit. Done this way, ai testing agents give you the one thing manual authoring never did, a test that was verified against the live page before it ever reached your branch.

6. Building a self-healing agent, and the Healer trap

Every team running a large end-to-end suite eventually wants the same thing: a bot that wakes up when a test goes red, works out what broke, and fixes it before anyone is paged. That is the self-healing agent, and it is the single most requested pattern in the whole conversation about ai agents for qa. It is also the one most likely to quietly destroy the value of your suite. This section builds a healer that is safe to run, and spends most of its words on why an unsafe healer is worse than no healer at all.

What a healer actually does

A healer is a loop wrapped around your existing runner. It watches for a failing test, gathers evidence (the error message, the stack, the Playwright trace, a DOM snapshot, recent git history), proposes a change, applies it on a branch, re-runs, and opens a pull request if the test is green again. Structurally it is the same plan-act-observe loop as every other of the ai testing agents in this guide. The difference is blast radius: this agent edits your test code, the one artifact you rely on to tell the truth about the product. That is what makes healers the sharpest edge in all of agentic testing.

async function heal(test) {
  let result = await run(test);
  while (result.status === "failed") {
    const patch = await llm({
      goal: "make this test pass",   // the entire trap lives in this line
      error: result.error,
      code: test.source,
    });
    await apply(patch);
    result = await run(test);
  }
  return patch;
}

Read that loop closely. The reward is one word, "pass," and the model will find the cheapest path to it. The cheapest path is almost never fixing the product.

The Healer trap: paid to lie

Here is the failure that turns a helpful agent into a liability. A checkout test asserts that the order total equals the sum of line items plus tax, a number you hand-verified when you wrote it. Someone ships a pricing bug that applies a discount twice, so the rendered total is wrong. The test correctly fails. A healer rewarded for green does not know or care that the app is broken. It sees a mismatch and does the obvious thing: it edits the expectation to match reality.

// checkout.spec.ts - the test is correct, the app has a bug
await expect(page.getByTestId("order-total"))
  .toHaveText("$126.00");   // 3 items + tax, verified by hand

// Actual rendered value: "$113.40" (discount applied twice)
// A "make it pass" agent rewrites the oracle:
await expect(page.getByTestId("order-total"))
  .toHaveText("$113.40");   // green, and the pricing bug now ships

The suite is green, the dashboard is calm, and you are undercharging every customer. This is not a hypothetical edge case, it is the default behavior of any healer whose objective is a passing run. The more capable the model, the more elegantly it will hide the defect. That is why unconstrained autonomous test agents are dangerous in exactly the situations where you most need them to be honest.

Classify the root cause before you touch anything

The fix is to split the job in two. First, a classifier decides why the test failed. Only then, and only for a narrow set of causes, does the healer get to edit. The classifier is where the real engineering lives, and it leans on how Playwright reports failures. A locator that "resolved to 0 elements" while the DOM snapshot clearly contains a renamed match is selector drift: test-side, safe to repair. A value mismatch where the locator resolved fine is an app regression: stop, do not touch the test, file a bug. This is the line that separates useful ai agents for qa from expensive bug-concealment machines.

type Cause = "selector_drift" | "timing" | "app_regression" | "unknown";

function classify(f: Failure): Cause {
  // element never resolved, but the DOM snapshot has a strong candidate
  if (f.error.includes("resolved to 0 elements") && domHasCandidate(f))
    return "selector_drift";      // test-side: eligible to heal

  if (f.error.includes("Timeout") && f.network.some(r => r.pending))
    return "timing";              // test-side: widen a wait, no value change

  // locator resolved, the asserted VALUE differs
  if (f.error.includes("toHaveText") && f.locatorResolved)
    return "app_regression";      // STOP. file a bug. do not edit the test.

  return "unknown";                 // STOP. escalate to a human.
}

The guardrails

Classification tells the agent what kind of failure it is looking at. The guardrails decide what it is physically allowed to do about it, and they are enforced by the harness, not requested in the prompt. A model can be talked out of a rule; a diff gate cannot. These rules are what let you point autonomous test agents at a live suite and sleep at night.

// healer.policy.ts - enforced by the runner, not the model
export const policy = {
  runsOn: ["selector_drift", "timing"],  // labeled, healable failures only
  maxChangedLines: 8,
  maxChangedFiles: 1,
  editableGlobs: ["**/*.spec.ts"],
  frozenGlobs:   ["src/**"],          // never touch application code
  forbid: [
    "delete-assertion",
    "change-expected-value",      // toBe / toHaveText / toEqual args are frozen
    "relax-matcher",              // toHaveText -> toContainText is a downgrade
    "add-skip-or-only",
  ],
  requireGreenReruns: 3,            // flake guard: pass 3x, not once
  merge: "human-reviewed-pr",        // never a direct push to main
};

Notice the pattern shared with the price-total trap. Every forbidden operation is a way to reach green without repairing anything. Freezing them removes the shortcut, which forces the agent to either do real work (repoint a drifted locator) or admit defeat (file a bug). Good agentic testing is mostly the discipline of removing cheap wins.

A safe heal, end to end

Here is what the whole loop looks like when it works. On the practice pages at app.thetestingacademy.com/playwright, a front-end refactor renames a hook from order-cta to checkout-cta. Behavior is identical, only the test id moved. The classifier sees "resolved to 0 elements" plus a strong DOM candidate, labels it selector drift, and hands it to the healer. The healer produces a one-line diff, well inside the diff cap, touching no assertion.

// data-testid renamed order-cta -> checkout-cta on the practice page
// app.thetestingacademy.com/playwright  (behavior unchanged, only the hook)
- await page.getByTestId("order-cta").click();
+ await page.getByTestId("checkout-cta").click();
// 1 file, 1 line, zero assertions touched -> within policy -> PR opened for review

It re-runs three times, stays green, and opens a pull request tagged with the trace and the classifier's reasoning. A human skims a one-line diff and merges. That is the entire promise of ai agents for qa delivered without a single weakened check.

7. Orchestrating a multi-agent QA crew

The instinct when you first reach for AI agents for QA is to write one enormous prompt: "here is my app, explore it, write a full Playwright suite, run everything, and fix whatever fails." It demos beautifully and rots within a week. A single agent holding the whole job drags a bloated context, emits a wall of code no reviewer wants to read, and collapses three very different risk profiles into one blast radius. The pattern that survives contact with a real codebase is a crew: small, single-purpose AI testing agents that pass typed artifacts to one another, with a human gate on every arrow between them.

Three roles cover most of the ground. A Planner explores the target and writes a plan. A Generator turns one line of that plan into one test. A Healer takes one labeled failure and proposes one fix. No agent does another's job, and that single constraint is most of the design. This is the difference between a party trick and agentic testing you can actually run against production every night.

Here is the crew at a glance. The column that matters most is the last one.

Why splitting beats one do-everything prompt

Three properties fall out of the split, each fixing something the mega-prompt gets wrong.

• Focused context. The Generator never needs to know how you discovered a journey, only the journey itself and your conventions. Feeding it the exploration transcript, the full DOM, and forty existing test files just hands the model more ways to hallucinate a selector. A narrow agent with a narrow context is more accurate and far cheaper per token.
• Reviewable artifacts. Each agent's output is a discrete file you can read in two minutes: a plan, one spec, one diff. You approve the plan before a single line of test code exists, and you review a Healer's patch in isolation instead of hunting it inside a 900-line pull request.
• Different risk profiles. Exploring is read-only. Generating adds net-new files that break nothing already passing. Healing edits existing tests and can quietly weaken them. Merge those into one prompt and you either over-review the safe work or, far worse, under-review the dangerous work.

AGENTS.md: the shared charter

The three agents share one file that every run loads first. Call it AGENTS.md (Claude Code reads CLAUDE.md, Codex and Cursor read AGENTS.md; symlink one to the other so all your AI testing agents see the same rules). It is the constitution for the crew: base URL, framework, locator policy, auth strategy, and a hard list of forbidden actions. Without it, three agents invent three incompatible styles and your suite becomes a museum of everyone's favorite selector.

# AGENTS.md - shared charter for the QA crew

Base URL:   https://app.thetestingacademy.com/playwright
Framework:  Playwright + TypeScript (@playwright/test)
Locators:   getByRole > getByLabel > getByTestId. Never nth-child CSS or XPath.
Auth:       reuse storageState .auth/user.json. Do not log in inside a test.
Oracles:    assert on user-visible text and roles, never on network internals.
Waiting:    auto-waiting only. page.waitForTimeout is banned.

# Forbidden, all agents
- Never edit files under src/ or api/.
- Never commit secrets or .env files.
- Never weaken or delete an assertion to make a test pass.

Keep it short and imperative. The charter is not documentation for humans, it is a set of rails, and every line you add is a line each agent must respect on every task.

The Planner: explore, then write a plan

The Planner drives the app through a Playwright MCP server (or a scripted crawl) and produces a structured plan, one entry per journey. Point it at the practice pages at app.thetestingacademy.com/playwright and it walks the login form, the table filters, and the drag-and-drop widget, recording what it saw. Critically, it writes down oracles, the observable thing that proves a journey worked, not just the steps to get there. It is forbidden from writing test code, because a plan you can read and prune is worth far more than code you have to reverse-engineer.

// plan.md entry - written by the Planner, pruned by a human
{
  "journey": "login-happy-path",
  "preconditions": ["clean storageState", "seed user [email protected]"],
  "steps": ["goto /playwright/login", "fill Username + Password", "click Sign in"],
  "oracle": "heading 'Welcome' visible AND url contains /dashboard"
}

You edit this file by hand. Delete journeys you do not want, tighten a vague oracle, add a precondition the agent could not know. The pruned plan is the contract the next agent must honor.

The Generator: one journey, one spec

The Generator reads AGENTS.md plus exactly one plan entry and emits one spec file. Because it sees a single journey, its context stays tight and its selectors stay honest: it uses the roles and labels the charter mandates rather than brittle CSS. It is forbidden from touching product code and from editing any other journey's spec, so a bad generation can never cascade into work you already trusted.

import { test, expect } from '@playwright/test';

test('login happy path', async ({ page }) => {
  await page.goto('https://app.thetestingacademy.com/playwright/login');
  await page.getByLabel('Username').fill('[email protected]');
  await page.getByLabel('Password').fill('Secret123!');
  await page.getByRole('button', { name: 'Sign in' }).click();
  await expect(page.getByRole('heading', { name: 'Welcome' })).toBeVisible();
});

You run the spec yourself. Green on the first try is common for a happy path against a stable page; when it is red, that is a signal for the third agent, not a license for the Generator to keep flailing at it.

The Healer: one labeled failure at a time

The Healer is where naive autonomous test agents do the most damage, because the lazy win is to make red tests green by gutting assertions. A disciplined Healer does the opposite. It takes exactly one failing test, its error, and its trace, then classifies the failure before proposing anything: is this selector drift where the app still behaves correctly, or is the app genuinely broken? Only the first kind earns a patch. The second gets a bug report, and the test stays red on purpose.

# Case A - LABEL: test-fix (selector drift, product behavior unchanged)
- await page.getByRole('button', { name: 'Sign in' }).click();
+ await page.getByRole('button', { name: 'Log in' }).click();

# Case B - LABEL: product-bug (do NOT patch the test)
# Sign in stays disabled after valid input. Filed BUG-214.
# Test left red on purpose. This is a real defect, not drift.

That classification is the load-bearing rule of the whole crew and the one thing that separates trustworthy agentic testing from a suite that lies to you. A Healer that edits product code, weakens an assertion, or heals more than one failure per run is not maintaining your tests, it is laundering real defects into green checkmarks.

The handoff: plan -> spec -> heal

How you run the crew is a detail: three subagents dispatched by an orchestrator, three terminal tabs, or three scheduled jobs. What is not a detail is that the agents never talk to each other directly. They hand off through files on disk, and the human review gate lives on each arrow.

Each row above is a place to stop. The artifacts are what make stopping cheap: a plan you approve, a spec you run, a diff you merge. Contrast that with the do-everything prompt, whose only artifact is a giant branch and a claim that it all passes. Narrow jobs, a shared charter, and a human on every handoff turn a pile of clever autonomous test agents into a QA crew you can actually trust with your suite.

8. Guardrails, review, security, and cost

An agent that writes and runs tests is a fast junior who never tires and never gets embarrassed by a green suite that proves nothing. The output of ai agents for qa looks like tests, reads like tests, and lands in your repo like tests, but a real fraction of it passes whether the app works or not. This section is the review gate, the team rules, the security posture, and the cost controls that turn agentic testing from an impressive demo into something you let near a codebase with 2,700 paying students behind it.

Review what the agent wrote, not just that it passed

Passing is not evidence. A model optimizing for a green run will reach for the shortest path to green, and a hollow assertion is shorter than a correct one. So generated suites fail in a recognizable way: the test drives the app but the check is empty, so the result is green no matter what the app did. Read every AI-authored test against the shape of the failure, not the color of the badge. Four patterns account for most of the rot.

Assertions that assert nothing. A bare page.locator('.total') builds a locator and runs no check; expect(page.url()).toBeTruthy() asserts that a string is a string. Both are green forever. Swallowed failures. A try/catch (or a trailing .catch(() => {})) around an expect turns a real failure into a log line, and the test still exits zero. Conditional assertions. An expect nested inside if (await ...isVisible()) is skipped whenever the branch is false, which is exactly the moment you wanted it to fail. Brittle nth chains. Positional locators like .nth(6).locator('td').nth(2) or absolute XPath survive until the first DOM shift, then flake for reasons no one can reproduce.

// 1. Asserts nothing: a bare locator runs no check
await page.locator('.cart-total');
expect(page.url()).toBeTruthy();            // a string is always truthy

// 2. Swallowed failure: the catch turns red into green
try {
  await expect(page.getByText('Order placed')).toBeVisible();
} catch (e) {
  console.log('banner missing, moving on');       // test still exits 0
}

// 3. Conditional assertion: skipped when the branch is false
if (await page.getByRole('alert').isVisible()) {
  await expect(page.getByRole('alert')).toContainText('Saved');
}

// 4. Brittle nth chain: positional, breaks on any DOM shift
await page.locator('div').nth(6).locator('td').nth(2).click();

The rewrite is what you require in review: web-first assertions on the thing under test, and role, label, or test-id locators that outlast a layout change. Point the agent at a stable target so the fix is easy to demonstrate, for example the practice checkout on app.thetestingacademy.com/playwright.

// Web-first assertion: auto-waits and fails loudly
await page.goto('https://app.thetestingacademy.com/playwright');
await expect(page.getByText('Order placed')).toBeVisible();

// Role / label / test-id locators survive layout churn
await page.getByRole('cell', { name: 'INR 1,499' }).click();
await expect(page.getByTestId('cart-line')).toHaveCount(3);

Team guardrails

Label every AI-authored change. Add a git trailer such as Assisted-by: playwright-agent or a commit-message prefix, so the batch is greppable and, when a bad run slips through, revertable in one command. You cannot audit or trust-scope what you cannot filter.

Quarantine until a human signs off. New agent tests land tagged and non-blocking, never on the PR gate that can hold up a human's merge. Run them nightly to gather flake signal, and promote a test out of quarantine only after someone has read it against the checklist above.

// AI-authored tests carry a tag until a human promotes them
test('@quarantine guest checkout applies coupon', async ({ page }) => { /* ... */ });

// PR gate runs everything except quarantine; nightly runs only those
npx playwright test --grep-invert @quarantine

Measure coverage of journeys, not test count. Autonomous test agents inflate test count with near-duplicate, shallow specs, so raw count rewards exactly the wrong behavior. Keep a named list of critical journeys (sign in, enroll and checkout, resume a course, submit a coding challenge, reset a password) and require each to have one green end-to-end path. A PR that adds twenty tests but leaves password reset uncovered has not moved the number that matters. This is the metric that keeps agentic testing honest.

Security: the agent takes real actions

A browser agent clicks real buttons, submits real forms, and sends real emails. That changes the threat model from "bad code" to "an unattended actor with a login." Five rules keep autonomous test agents boxed in.

Never point them at production. Staging or an ephemeral per-run environment only. Give them a throwaway account: least-privilege, seeded, and safe to burn, never an admin or a real student. Allowlist origins so the browser cannot wander onto a payment gateway or an external link. Load secrets from the environment (process.env, a Playwright storageState file, a secrets manager), never pasted into the prompt where they land in transcripts and model logs. Treat page content as untrusted: the DOM the agent reads can carry instructions aimed at the model.

Cost: tokens per step add up fast

Every agent step resends context (the current page snapshot plus the running history) to the model, so cost grows with session length, not with the number of assertions you get. Two levers keep the bill sane. Scope sessions: one journey per session, cap the step budget so a stuck agent cannot loop forever, and reuse a saved storageState to skip re-login on every run. Prefer snapshots over screenshots: browser-tool layers like the Playwright MCP server default to a text accessibility-tree snapshot and expose screenshot and coordinate tools only as an opt-in capability (--caps vision). The text snapshot costs a fraction of the tokens an image input does, and models act on structured text more deterministically than on pixels. Reserve vision for the rare case where you genuinely need to judge layout or a canvas. Handled this way, ai testing agents stay cheap enough to run on every pull request instead of once a quarter.

9. Roadmap recap and FAQ

You now have the moving parts: the tool layer, the oracle, the guardrails, the review loop. This closing section compresses everything into a maturity roadmap you can act on this week, then answers the six questions every team asks once ai testing agents stop being a demo and start touching real pipelines. Treat it as an adoption plan for agentic testing, not a summary.

The maturity roadmap, recapped

Adopt this in order. Each stage earns the trust budget for the next, and skipping ahead is how ai agents for qa end up quarantined after one bad week. Score your team honestly against these five stages before you widen scope.

1. Assistant. The agent drafts locators, assertions, and page objects; a human reviews and commits every line. Deterministic execution, zero autonomy at runtime. Every team should start here.
2. Generator. Given a grounded prompt, the agent writes one complete test from the real accessibility tree, then stops for review before it merges. Coverage compounds without the boilerplate tax.
3. Healer. When a selector breaks, the agent proposes a repair from the current DOM or accessibility tree and opens a pull request; humans still merge. Flake and maintenance hours drop first at this stage.
4. Multi-agent. A planner splits the work across specialized agents (explorer, generator, healer) under one orchestrator and a shared tool set, coordinating through a task queue instead of a single mega-prompt.
5. Autonomous. Under guardrails, the agents own coverage of defined journeys inside CI, gated by budgets and by human approval on anything that writes. This is the ceiling, not the entry point.

Do agents replace SDETs?

No, and the framing is wrong. What changes is where your hours go. Instead of hand-writing every wait and assertion, you design the harness the agent operates inside: the tools it can call, the oracle that decides pass or fail, the guardrails that bound its blast radius, and the review gate on its output. Those are senior skills, and they are in shorter supply than test scripts. An agent with no human-defined notion of "correct" will confidently automate the wrong behavior. The role shifts from author to supervisor of ai agents for qa: you curate tools, tune prompts, read traces, and own the judgment calls a model cannot make. Teams that adopt agentic testing well tend to raise the bar on their engineers rather than cut headcount, because leverage per person goes up. The repetitive authoring shrinks; the design, review, and reliability work grows.

Which framework should I start with?

Start where your CI, fixtures, and test data already live. If you are greenfield, Playwright is the pragmatic default for agent work: it exposes a clean accessibility snapshot the model can read, ships an official MCP server so an agent can drive a browser through structured tool calls, and its trace viewer makes each run auditable after the fact. The practice pages at app.thetestingacademy.com/playwright are a safe, stable target to prototype against before you point anything at your own app. Selenium teams do not need to rip and replace: wrap the existing Grid behind an MCP or a thin tool layer and let the agent call it. The framework matters less than the interface you expose to the model. Pick the stack your team can debug at 2 a.m., because you will.

How do I stop an agent breaking my app?

Bound the blast radius before you grant any autonomy. autonomous test agents should run against ephemeral or staging environments seeded with disposable data, never production. Scope every tool: read-only database roles, an API allowlist that excludes destructive verbs, a domain allowlist so the browser cannot wander off. Put a human approval gate on any action that writes state, and cap tokens and runner calls so a runaway loop burns a budget, not your infrastructure. Log every tool call so a bad run is reconstructable. The mental model is least privilege plus a kill switch: assume the agent will eventually do the dumbest thing its permissions allow, and make sure that thing is harmless.

Can it test authenticated flows?

Yes, and this is where most real coverage lives. The clean pattern is to mint a session once and hand the agent the token, never the raw credentials. In Playwright, a setup project logs in with a dedicated test account and saves storageState; every agent-driven test then reuses that state and starts already authenticated.

// auth.setup.ts - mint a session once, reuse it everywhere
import { test as setup } from '@playwright/test';

setup('authenticate', async ({ page }) => {
  await page.goto('https://app.thetestingacademy.com/playwright/login');
  await page.getByLabel('Email').fill(process.env.TEST_EMAIL!);
  await page.getByLabel('Password').fill(process.env.TEST_PASSWORD!);
  await page.getByRole('button', { name: 'Sign in' }).click();
  await page.waitForURL('**/dashboard');
  // give the agent the saved session, not the secrets
  await page.context().storageState({ path: '.auth/agent.json' });
});

For SSO or MFA, prefer a programmatic login against your auth API to issue a session directly, or a test-only bypass on staging. Keep real user credentials, production accounts, and second factors out of the agent's reach entirely. The agent needs a valid session, not your secrets.

How many agents do I need?

Fewer than you think. Start with one. A single orchestrator with a sharp tool set and a clear oracle outperforms a swarm of loosely defined agents that duplicate work and multiply cost. Add a second agent only when one agent's context genuinely overflows, for example when exploration, authoring, and triage each need their own focused instructions and history. Even then, three specialized roles under one orchestrator is plenty for most suites. Every extra agent adds coordination overhead, token spend, and one more thing to debug when a run goes sideways. Agent count is a cost, not a feature: justify each one with work a single agent measurably could not do.

How do I measure if it is working?

Not by tests generated, which is a vanity metric an agent can inflate in an afternoon. Measure outcomes: escaped defects caught before release, flake rate of the agent-authored suite, maintenance hours saved, mean time to author a covered journey, and the false-positive rate of triage (how often the agent cries bug on a flake). Track cost per run so a metered model stays honest. If ai testing agents are working, escaped bugs and maintenance hours fall while trusted coverage of critical journeys rises, at a cost per test you can defend. If the only number going up is generated test count, you have built a demo, not a capability.

Where to go next

Two capabilities turn a competent agent into a reliable one. MCP for QA standardizes how your agent reaches tools, browsers, test runners, databases, and CI through one typed interface, so you stop gluing bespoke scripts together. RAG for QA grounds the agent in your own requirements, test cases, and past bug reports, so it stops guessing your domain and starts citing it. Read those two next: they are the difference between ai agents for qa that impress in a demo and autonomous test agents you would actually ship behind.

Series siblings: MCP for QA and RAG for QA. See also Prompt vs Skill vs Agent, LLM vs AI Agent, and the Playwright MCP guide.