Codex
for
QA.

The room you just walked into

QA is no longer the last step.
It is the agent's quality system.

Where Codex sits in your stack.

Think of Codex as a harness-native pair tester. It has repo context, shell access, patch-based edits, web search, image inputs, browser/computer use through the app, MCP connectors, and review mode. Your advantage is turning that into a repeatable QA factory.

Four surfaces. One workflow.

Setup · macOS / Linux / Windows

Five lines. One terminal.

# 1. install Codex CLI
npm i -g @openai/codex

# 2. open your QA repo
cd ~/work/qa-portfolio

# 3. start Codex
codex

# 4. first run: sign in when prompted
# ChatGPT account or API key auth

# 5. scaffold repo guidance
> /init  → creates AGENTS.md

A session has four moving parts.

Three ways to steer Codex.

> run checkout smoke, fix only locator flakes,
  then show the diff and test output

> /plan
> /model
> /review

npx playwright test --grep @smoke
npm run lint
git diff --stat

The complete QA cheat-sheet

Every slash command a QA touches.

QA habit · type /plan before broad changes, /diff before claiming done, and /review before opening a PR.

Patch · Shell · Search · Browser.
Everything else is orchestration.

> find the flaky wait in tests/login.spec.ts,
  replace it with an assertion-based wait,
  run that spec 20 times, and report the pass rate.

// Codex will usually:
  rg(waitForTimeout|sleep|networkidle)
  apply_patch(tests/login.spec.ts)
  shell(npx playwright test tests/login.spec.ts --repeat-each=20)
  summarize(diff + exit code + failures)

Give Codex just enough autonomy.

What goes inside AGENTS.md

Nine sections every QA repo needs.

## 1. Stack
Playwright, TypeScript, APIRequestContext, axe.
## 2. Commands
pnpm lint · pnpm test · npx playwright test.
## 3. Folder layout
tests/e2e · tests/api · tests/fixtures · tests/pom.
## 4. Locators
Role/label/testid first. XPath forbidden.
## 5. Waits
No hard sleeps. Use assertions and expect.poll.
## 6. Test data
No real PII. Use builders and env-based users.
## 7. Tags
@smoke @regression @a11y @visual @flaky.
## 8. Review
Prioritize bugs, regressions, missing tests.
## 9. Guardrails
Ask before deleting specs, changing CI, or bumping deps.

Context is a budget. Spend it like one.

Gemini, OpenAI-compatible endpoints, and local models

Use Gemini as a second expert, not a random swap.

# install
npm install -g @google/gemini-cli

# run in the same repo
gemini

# pick a specific Gemini model
gemini -m gemini-2.5-flash

# non-interactive review
gemini -p "Review tests/e2e for missing assertions" \
  --output-format json

# ~/.codex/config.toml
model_provider = "qa-gateway"
model = "gemini/gemini-2.5-pro"

[model_providers.qa-gateway]
name = "QA model gateway"
base_url = "https://gateway.example.com/v1"
env_key = "QA_GATEWAY_API_KEY"
wire_api = "responses"
supports_websockets = false

Important · Codex custom providers currently use the Responses protocol. Google's Gemini OpenAI-compatibility examples use Chat Completions, so direct Gemini endpoint routing may not be enough unless your gateway translates to Responses. For practical QA teams, native Gemini CLI plus Codex is the reliable path.

A tester's routing table.

Spawn specialists, not chaos.

Codex can spawn specialized agents in parallel when you explicitly ask. Built-ins include default, worker, and explorer. Custom agents live as TOML files under ~/.codex/agents/ or .codex/agents/.

> Review this branch vs main. Spawn one agent per topic:
  1. security risk
  2. test flakiness
  3. missing assertions
  4. API contract risk
  5. maintainability
Wait for all agents, then summarize the top 8 findings.

Skills are playbooks Codex loads on demand.

A skill is a directory with a required SKILL.md plus optional scripts/, references/, assets/, and helper files. Codex starts with the skill name and description, then reads the full instructions only when the task matches.

---
name: flake-hunter
description: Use when a Playwright spec fails intermittently or contains waitForTimeout, sleep, networkidle, or brittle locator patterns.
---

# Flake Hunter
1. Read the failing spec and related fixture.
2. Search for hard waits and brittle selectors.
3. Replace with role locators and assertion waits.
4. Run the spec with --repeat-each=20.
5. Report pass rate, changed lines, and remaining risk.

Build your own Codex for QA

Package your testing brain as skills.

> Use $skill-creator to create a repo-scoped skill named locator-auditor.
It should trigger when tests use XPath, CSS chains, nth-child, test-only waits,
or missing assertions. It should read tests/, output a risk table, and only edit
when I explicitly say "fix them".

When a skill is not enough, use a plugin.

Plugins package skills, MCP servers, and apps together. For a QA organization, a plugin can ship the company browser tools, Jira connector, test-data service, and house skills as one installable bundle.

Hooks fire around tool calls.

Hooks let you run scripts on Codex lifecycle events: prompt submit, pre-tool, permission request, post-tool, compaction, subagent start/stop, session start, and stop. Use them for formatting, test targeting, audit logs, and safety gates.

"PostToolUse": [{
  "matcher": "apply_patch",
  "hooks": [{
    "type": "command",
    "command": "npm run lint -- --quiet"
  }]
}],
"PreToolUse": [{
  "matcher": "shell",
  "hooks": [{
    "type": "command",
    "command": "node scripts/block-main-edits.js"
  }]
}]

MCP turns Codex into a browser, Jira, GitHub, docs, and your internal API.

Model Context Protocol exposes external tools as structured actions. For QA, the high-value servers are Playwright/browser, Jira/Confluence, GitHub, design systems, test data, and custom product APIs.

codex mcp add playwright -- npx @playwright/mcp@latest

A QA agent needs eyes and hands.

Use the in-app browser for local apps, screenshots, accessibility snapshots, and visual verification. Use Chrome automation when cookies, extensions, or logged-in remote sessions matter.

Demo · authoring a spec without typing locators

"Open saucedemo.com, log in, add an item, screenshot the cart."

import { test, expect } from '@playwright/test';

test('adds backpack to cart', async ({ page }) => {
  await page.goto('https://www.saucedemo.com');
  await page.getByRole('textbox', { name: /user/i }).fill('standard_user');
  await page.getByRole('textbox', { name: /pass/i }).fill('secret_sauce');
  await page.getByRole('button', { name: 'Login' }).click();
  await page.getByRole('button', { name: /add to cart/i }).first().click();
  await expect(page.getByText('Sauce Labs Backpack')).toBeVisible();
});

API tests from a single curl.

Paste a curl, OpenAPI URL, or Postman export. Codex can infer happy path, negative path, schema validation, auth variants, and fixture structure.

> generate Playwright API tests for this endpoint.
Include positive, invalid email, missing auth, schema,
and one contract drift check. Use zod for runtime validation.

curl -X POST https://api.demo.dev/v1/users \
  -H 'Authorization: Bearer $T' \
  -H 'Content-Type: application/json' \
  -d '{"email":"[email protected]","plan":"pro"}'

tests/api/users.spec.ts
fixtures/apiClient.ts
tests/contracts/user.schema.ts

Coverage:
- 201 create user
- 400 invalid payload
- 401 missing token
- schema validation
- idempotency or duplicate email behavior

From a screenshot to a filed bug.

> [screenshot attached]
User reports the price chip overflows on mobile.
Reproduce at 390x844, capture screenshot + trace,
find likely component, and draft a Jira-ready bug
with steps, expected, actual, severity, and evidence.

QA reviews code too. Now they have leverage.

tests/login.spec.ts:14 · P1
page.waitForTimeout(2000) hides race.
Fix: wait for dashboard heading and toast.

src/auth/middleware.ts:42 · P0
Token branch skips expiry validation.
Fix: assert exp before session creation.

playwright.config.ts:8 · P2
retries: 3 masks flakes.
Fix: retry once, quarantine with owner.

Run Codex in your pipeline.

Use headless Codex only for narrow, bounded CI jobs: summarize failing tests, draft PR review comments, triage smoke failures, or create a follow-up issue. Keep prompts small and permissions tight.

name: qa-bot
on: { pull_request: { types: [opened, synchronize] } }
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Codex QA review
        env: { OPENAI_API_KEY: secrets.OPENAI_API_KEY }
        run: |
          codex exec --model gpt-5.4-mini \
            "Review this PR for QA risk only. Focus on flaky waits,
             hard-coded data, missing assertions, and broken smoke coverage.
             Output max 8 bullets with file:line when possible."

Plan first. Then let it loose.

Together these are the two controls that let you delegate real work without losing engineering judgment.

Things that will save you a workday.

A QA day · before vs after.

When one repo has 12 apps and 8 suites

Codex in a monorepo.

Do not rewrite by hand

Selenium · Cypress · TestCafe → Playwright.

Let Codex inspect the old suite, build a migration matrix, convert one vertical slice, run it, then scale. Ask Gemini to independently review missing behavior before deleting the old tests.

> Migrate cypress/e2e/checkout to Playwright under tests/e2e/checkout.
Rules: prefer getByRole/getByLabel/getByTestId, no waitForTimeout,
cy.intercept becomes page.route, fixtures become typed builders.
Convert one spec first, run it, show diff and pass/fail before continuing.

Numbers that drive QA decisions

Flake rate · p95 · MTTR — let Codex do the math.

Capstone · build your own Codex for QA

Skill · model routing · MCP · site · live URL.

Step 1 · foundation

Create AGENTS.md and the first skill.

> Build the QA agent foundation for this repo.
Create AGENTS.md with commands, locator policy, wait policy, data policy,
review rules, and stop conditions. Then create .agents/skills/locator-auditor/SKILL.md.
Do not edit product code. Run the skill on tests/ and report the top 10 risks.

Step 2 · model routing

Codex edits. Gemini critiques.

> Add a docs/model-routing.md file for our QA team.
Include when to use gpt-5.5, gpt-5.4, gpt-5.4-mini, Gemini CLI,
and local models. Add examples for: flaky spec, API contract suite,
release-readiness review, and migration planning. Keep it practical.

Step 3 · Playwright + CI

Cover every route. Gate every merge.

> Add Playwright smoke, a11y, visual, link, and SEO suites.
Run mobile 390x844 and desktop 1440x900. Add CI with lint, typecheck,
e2e, and report upload. If anything fails locally, fix it before reporting done.

Step 4 · deploy

From localhost to thetestingacademy.com.

> Deploy ./qa-portfolio to Vercel.
Run the Playwright suite against the prod URL.
Print preview URL, prod URL, test report path, and next manual checks.

> Publish this Codex masterclass page under
app.thetestingacademy.com/masterclass/codex.html.
Verify with curl and a browser snapshot after deploy.

Don't do these. Ever.

The new QA toolbelt.

Where to go next.