Kiro lessons for QA.

Kiro's Spec-Driven QA Model

Kiro specs connect requirements, design, tasks, and implementation. QA adds risk, observable acceptance, counterexamples, and an independent definition of done.

A spec is useful when each artifact narrows ambiguity without erasing important risk. Requirements state behavior, design explains the system decision, tasks create an executable plan, and tests prove selected outcomes. None of those files certifies the others merely because Kiro generated them together.

Choose the lightest workflow that preserves the needed evidence. A feature spec suits a new cross-file capability. A bugfix spec starts from observed incorrect behavior. Quick Plan suits a contained change whose risk and design are already understood. High-impact authentication or payment changes deserve the deeper chain.

Workspace Setup, Privacy, and Trust Boundaries

Kiro can read project context, execute tools, run hooks, and connect MCP servers. Establish file, command, network, and credential boundaries first.

Open a disposable repository and review project trust, privacy settings, enabled tools, hooks, and MCP configuration. Keep customer data and production credentials out of the workspace. Use a synthetic application whose test command is deterministic on a clean checkout.

Kiro CLI V3 introduces capability-based permissions and standalone hooks; older embedded-hook or trust-all examples may not match the current engine. Record the surface you are teaching and review current documentation before copying configuration between IDE and CLI.

git switch -c training/kiro-lockout
git status --short
npm ci
npx playwright test tests/auth/lockout.spec.ts --project=chromium

# CLI V3, when used:
kiro-cli --v3
# Inside the session: /spec new account-lockout

Trust review

• Which paths may the agent read and write?
• Which commands may execute without a separate approval?
• Which MCP servers can send data outside the workstation?
• Which hooks can block, mutate, or publish after a task?
• Which person reviews the final diff and residual risk?

Steering Files and AGENTS.md for Test Standards

Steering gives Kiro durable project context. Inclusion rules keep specialized guidance attached only when the related files or tasks need it.

Store project-specific guidance in .kiro/steering/. Use always-included content sparingly for repository-wide facts. Use file-match guidance for Playwright, API, mobile, or performance conventions. Manual inclusion is useful for deep references that should enter context only on request.

Kiro also recognizes AGENTS.md. Avoid duplicating conflicting rules across files. State where a standard is authoritative, who owns it, and which deterministic command enforces it. Steering can describe locator policy; lint and tests must still catch violations.

---
inclusion: fileMatch
fileMatchPattern: "tests/**/*.spec.ts"
---

# Playwright quality contract

- Trace every critical test to a requirement ID.
- Prefer role, label, or test-id locators.
- Never add fixed sleeps, force clicks, or silent retries.
- Preserve assertion strength and test count.
- Verify with lint, typecheck, and the focused Chromium project.

EARS Requirements With Observable Acceptance

EARS patterns turn vague behavior into trigger, condition, system response, and measurable outcome while keeping policy ownership visible.

Use ubiquitous requirements for behavior that always applies, event-driven requirements for triggers, state-driven requirements for a condition, and unwanted-behavior requirements for failures and abuse. The template is a clarity tool, not a substitute for domain review.

For login lockout, define threshold, counting window, lock duration, response privacy, audit event, and recovery. Avoid implementation details such as database column names unless they are genuine constraints. Every requirement needs an observable oracle available to the test level that owns it.

### AUTH-LOCK-01 - Threshold
WHEN a known account receives five consecutive invalid-password attempts within ten minutes,
THE authentication service SHALL reject subsequent password attempts for fifteen minutes.

### AUTH-LOCK-02 - Privacy
IF an account is locked,
THEN the public response SHALL remain indistinguishable from the normal invalid-credential response.

### AUTH-LOCK-03 - Recovery
WHEN the lock period expires,
THE authentication service SHALL accept a valid credential without administrator action.

Feature Specs: Requirements, Design, and Tasks

A feature spec creates a linked set of artifacts. QA reviews the transitions between them for missing risks, untestable design choices, and tasks with no evidence.

Kiro stores a spec under .kiro/specs/<name>/. Review requirements.md before accepting design, and review design.md before accepting tasks. Regeneration can change meaning, so keep the files in Git and inspect diffs rather than relying on the current chat summary.

Design should expose test seams: clock injection for lock expiry, an audit sink, stable API responses, and isolated account state. Tasks should include the test or evidence work next to the implementation, not as an optional final cleanup item.

.kiro/specs/account-lockout/
  requirements.md
  design.md
  tasks.md

tests/auth/
  lockout.api.spec.ts
  lockout.e2e.spec.ts
  lockout.property.spec.ts

Bugfix Specs: Current, Expected, and Unchanged Behavior

A bugfix spec preserves the observed defect, the intended correction, and adjacent behavior that must not regress.

Start with a reproducible observation, not the suspected code change. Record environment, data, exact command, actual result, and expected result. Then name unchanged behavior that a narrow fix could accidentally damage. Kiro's bugfix workflow can turn those sections into a focused plan and regression properties.

For a lockout reset defect, unchanged behavior includes generic error responses, valid login before the threshold, separate counters per account, and normal expiry. This prevents a fix that merely disables lockout or changes public messages.

## Current behavior
After attempt 4 succeeds in failing, a page refresh resets the counter.
Reproduction: npx playwright test tests/auth/lockout-refresh.spec.ts

## Expected behavior
Refresh does not alter the server-owned failure count. Attempt 5 locks the account.

## Unchanged behavior
- Attempts 1-4 return the generic invalid-credential response.
- Another account keeps an independent counter.
- A valid password works after the approved lock period.

Regression properties

• Refreshing the client never decreases a server-owned failure count.
• No sequence shorter than the threshold creates a lock.
• No sequence at or above the threshold creates a session during the lock window.
• The public response does not reveal whether the account exists or is locked.

Traceability From Task to Playwright Evidence

Stable IDs and a small manifest connect the approved requirement and task to the test, command, report, and commit that provide evidence.

Put requirement IDs in test metadata or titles in a way your reporter can extract. Link tasks to those IDs and to the exact test command. A traceability check should fail when a critical requirement has no test, a test cites an unknown requirement, or a completed evidence task points to no report.

Traceability does not mean one test per sentence. One scenario may cover several related requirements, and a property test may cover a large state space. The important part is that a reviewer can follow the claim without asking the agent to reconstruct its reasoning.

test("AUTH-LOCK-01 locks on the fifth invalid attempt", {
  annotation: [
    { type: "requirement", description: "AUTH-LOCK-01" },
    { type: "risk", description: "SEC-P0-03" },
  ],
}, async ({ page }) => {
  // synthetic setup and observable assertions
});

{
  "requirement": "AUTH-LOCK-01",
  "task": "4.2",
  "test": "tests/auth/lockout.e2e.spec.ts",
  "command": "npx playwright test tests/auth/lockout.e2e.spec.ts",
  "report": "artifacts/lockout/results.json",
  "commit": "<git-sha>",
  "result": "passed"
}

Correctness Properties and Property-Based Testing

Example tests prove selected journeys. Properties express invariants across generated sequences and shrink a failure to a useful counterexample.

Kiro can derive candidate properties from EARS requirements. QA must review whether each property is observable, meaningful, and independent of the implementation. Pair properties with named examples; do not replace a critical user journey with an opaque generator.

For lockout, generate sequences of valid and invalid attempts, account identities, timestamps, and refresh events. Freeze the clock and isolate state so a failure is reproducible. Record the random seed and shrunk counterexample in the test report.

fc.assert(
  fc.property(fc.array(fc.boolean(), { maxLength: 12 }), async attempts => {
    const model = new LockoutModel({ threshold: 5 });
    const system = await createSyntheticAccount();

    for (const passwordIsValid of attempts) {
      const expected = model.attempt(passwordIsValid);
      const observed = await system.attempt(passwordIsValid);
      expect(observed.sessionIssued).toBe(expected.sessionIssued);
      expect(observed.publicMessage).toBe("Invalid email or password");
    }
  }),
  { seed: 20260715, numRuns: 100 }
);

Hooks That Run Tests and Block Unsafe Work

Kiro hooks execute agent prompts or shell commands on lifecycle events. Command hooks provide deterministic enforcement when the trigger can block.

Current standalone hooks live in .kiro/hooks/*.json with a versioned schema. Use PostTaskExec to run a focused check after a task. Use a blocking trigger such as PreTaskExec, PreToolUse, or UserPromptSubmit when exit code 2 must prevent unsafe execution. A nonblocking post hook can report failure but cannot retroactively undo side effects.

Keep commands fast, deterministic, and repository-owned. A hook that invokes an agent to 'make tests pass' is not equivalent to a hook that runs the fixed test command. Commit hook files and review matcher breadth, timeout, environment, and secret access.

{
  "version": "v1",
  "hooks": [
    {
      "name": "auth-test-on-save",
      "trigger": "PostFileSave",
      "matcher": "^(src/auth|tests/auth)/.*\\.(ts|tsx)$",
      "action": {
        "type": "command",
        "command": "npx playwright test tests/auth --project=chromium"
      },
      "timeout": 120,
      "enabled": true
    }
  ]
}

Playwright MCP With a Constrained Browser Target

A browser MCP server can inspect and operate a UI, but the test oracle must still come from requirements and deterministic assertions.

Connect a reviewed Playwright MCP server only to a local or synthetic environment. Give the agent a dedicated test account and no payment, administration, or customer-data privileges. Record which tools are exposed before the session and remove any capability the scenario does not need.

Use browser exploration to collect accessible names, network behavior, and candidate reproduction steps. Convert accepted observations into version-controlled Playwright tests. The MCP transcript is useful investigation evidence, but it is not a stable regression suite.

1. Start the synthetic target on a local port and reset its data.
2. Review the MCP server package, command, arguments, and network scope.
3. Allow navigation, snapshot, click, and input only for the exercise.
4. Reproduce lockout and compare the browser observation with API and audit evidence.
5. Encode the accepted flow as a deterministic test and rerun from a clean state.

Powers, Tool Authority, and Supervised Execution

A Kiro Power packages reusable instructions and integrations. Reuse magnifies both good standards and excessive permissions, so ownership and versioning matter.

A Power can include a POWER.md, MCP configuration, steering, and hooks. Build a QA Power around a narrow outcome such as reviewing Playwright changes or collecting a test-failure evidence bundle. Keep its required tools minimal and document every side effect.

Review Powers like code dependencies: source, version, maintainer, requested tools, network access, secrets, and update path. Do not treat a friendly name or generated description as proof of safety. For high-impact actions, require explicit approval and an external branch or environment policy.

# Playwright Failure Review

## Purpose
Classify one failing Playwright test and prepare a bounded repair proposal.

## Inputs
- Requirement ID and revision
- Raw failure and trace
- Allowed test paths

## Allowed actions
- Read repository and reports
- Run the focused test and lint
- Propose a test-only diff

## Prohibited actions
- Product edits, secret access, deployment, test deletion, assertion weakening

## Completion
Evidence bundle plus human review; never auto-merge.

Power review checklist

• Purpose and completion are narrower than the tool list.
• All MCP packages and commands have reviewed provenance.
• Secrets use least privilege and are absent from logs and prompts.
• Hooks cannot publish, deploy, or mutate protected systems silently.
• A maintainer and rollback path are named.

Capstone: Specify and Verify Login Lockout

Carry one security-sensitive behavior from EARS requirements through design, tasks, example tests, properties, hooks, evidence, and human acceptance.

The capstone uses a synthetic authentication service. It locks a known account after five consecutive invalid attempts within ten minutes, keeps public responses generic, isolates counters per account, and recovers after fifteen minutes. A frozen clock and reset endpoint exist only in the local test harness.

Do not begin implementation until a reviewer accepts requirement thresholds and observability. Do not mark tasks complete until the focused example tests, property tests, trace check, and hook command pass from a clean checkout.

.kiro/specs/account-lockout/
  requirements.md      # EARS IDs AUTH-LOCK-01..04
  design.md            # clock, counter, audit, response seams
  tasks.md             # implementation and evidence tasks
.kiro/steering/playwright-testing.md
.kiro/hooks/auth-test-on-save.json
tests/auth/lockout.api.spec.ts
tests/auth/lockout.e2e.spec.ts
tests/auth/lockout.property.spec.ts
artifacts/lockout/evidence.json

Acceptance sequence

1. Review the EARS requirements with security and product owners.
2. Trace every requirement to design and at least one evidence task.
3. Prove the original defect or missing behavior where applicable.
4. Run lint, typecheck, example tests, property tests, and traceability checks.
5. Trigger the committed hook and preserve its output.
6. Have another SDET reproduce the suite and record residual risks.