Loop Engineering lessons for QA.

Loop Engineering in QA

Loop Engineering designs the control system around a maker: goal, evidence, bounded action, independent verification, persistent state, and an explicit decision.

1. Choose a reversible artifact. Begin with a test, plan, case set, or report that can be restored and reviewed.
2. State the authority boundary. The maker may propose or edit. It may not redefine acceptance, approve release risk, or deploy.
3. Put verification outside the maker. Fixed checks must run independently and return structured results.
4. Name the human owner. A person accepts, revises, rejects, or stops the candidate.

Prompt, Context, Harness, Loop

These layers work together, but they solve different engineering problems. Mixing them produces systems that look autonomous without being controlled.

Goal and Observable Exit

A loop needs an outcome a verifier can observe. "Fix the test" is vague; "restore the login test without changing product behavior or weakening coverage" is enforceable.

1. Describe the intended behavior, not the preferred code edit.
2. Name every acceptance check and the required exit code.
3. Define accepted, rejected, blocked, and escalated states.
4. Set an attempt ceiling and a stagnation rule before execution.

{
  "goal": "Restore the login test without changing product behavior",
  "acceptWhen": [
    "policy.exitCode == 0",
    "targetTest.exitCode == 0",
    "immutableOracle.exitCode == 0"
  ],
  "acceptedState": "AWAITING_HUMAN_APPROVAL",
  "attemptCeiling": 3
}

Evidence Before Action

Give the maker enough evidence to support a narrow correction, but no secrets, irrelevant repository history, or authority it does not need.

Minimum useful evidence bundle

• The exact failing command, exit code, and focused output.
• The mutable artifact and the smallest related source surface.
• The protected behavior contract or acceptance oracle.
• The allowlist, denylist, stop rules, and prior attempt result.

{
  "failureId": "target-test/attempt-01",
  "exitCode": 1,
  "signal": "button named Login was not found",
  "mutablePaths": ["tests/login.spec.ts"],
  "protectedIntent": "A valid user reaches Dashboard",
  "secretsPresent": false
}

The Bounded Maker

The maker proposes the smallest correction inside an explicit surface. The controller, policy, oracle, and reviewer remain outside its authority.

1. Allow mutation only where a correction is expected.
2. Require a diagnosis, changed-path list, and rationale.
3. Reject any observed path outside the allowlist.
4. Restore from baseline before a new exercise or after rejection.

{
  "mutablePaths": [
    "tests/login.spec.ts",
    "pages/"
  ],
  "protectedPaths": [
    "app/",
    "tests/oracle/",
    "loop/contract.json",
    "scripts/"
  ],
  "mergeAuthority": false,
  "deploymentAuthority": false
}

State and Stop Decisions

Conversation memory is not control state. Persist attempts, verifier results, evidence paths, and the current decision in a file or database the controller owns.

READY
  -> OBSERVING
  -> MAKING
  -> VERIFYING
  -> RECORDING
  -> ITERATE | ESCALATE | AWAITING_HUMAN_APPROVAL

Stop when:
  acceptance passes
  attempt ceiling is reached
  identical failure repeats
  scope or policy is violated
  required evidence is unavailable

Keep a compact state record with the goal, current status, attempt entries, failed checks, maker response, reviewer verdict, and pending human decision. A resumed process reads that record rather than trusting chat history.

Machine-Readable Contract

The contract is the controller's source of truth. Prompts may explain it, but only structured fields should drive permissions, checks, and stop decisions.

{
  "schemaVersion": 1,
  "goal": "Restore the login test without changing product behavior or weakening coverage.",
  "mutablePaths": ["tests/login.spec.ts", "pages/"],
  "protectedPaths": ["app/", "tests/oracle/", "loop/", "scripts/"],
  "checks": [
    { "id": "policy", "command": "node", "args": ["scripts/check-policy.mjs"] },
    { "id": "target-test", "command": "npx", "args": ["--no-install", "playwright", "test", "tests/login.spec.ts", "--repeat-each=2"] },
    { "id": "immutable-oracle", "command": "npx", "args": ["--no-install", "playwright", "test", "tests/oracle/login-contract.spec.ts"] }
  ],
  "attemptCeiling": 3,
  "successState": "AWAITING_HUMAN_APPROVAL"
}

Contract review

• Every path is repository-relative and normalized.
• Commands and arguments are arrays, not concatenated shell strings.
• The success state requests review; it does not merge or deploy.
• The controller rejects unknown schema versions and missing fields.

Mutable and Protected Paths

An allowlist controls where a candidate may change files. A sealed baseline proves that protected files still match their original content.

1. Resolve every observed path against the repository root.
2. Reject traversal, symlink escape, and ambiguous path prefixes.
3. Hash protected files after reset and before acceptance.
4. Fail closed when a protected file is missing, new, or changed.

{
  "algorithm": "sha256",
  "files": {
    "app/login.html": "<sealed-hash>",
    "tests/oracle/login-contract.spec.ts": "<sealed-hash>",
    "loop/contract.json": "<sealed-hash>",
    "scripts/check-policy.mjs": "<sealed-hash>"
  }
}

Deterministic Verifier

A verifier runs fixed commands, captures outputs and exit codes, and emits a structured manifest. It does not ask the maker whether the work passed.

1. Load checks from the contract.
2. Spawn each command directly with an argument array.
3. Capture standard output, standard error, exit code, and signal.
4. Write one immutable log per check and a manifest for the attempt.
5. Return pass only when every required check passes.

{
  "id": "target-test",
  "command": "npx",
  "args": ["--no-install", "playwright", "test", "tests/login.spec.ts", "--repeat-each=2"],
  "exitCode": 0,
  "signal": null,
  "passed": true,
  "logPath": "loop/evidence/attempt-02/target-test.log",
  "logSha256": "<hash>"
}

Immutable Oracle

The target test can be wrong. The oracle independently preserves product intent and stays outside the maker's editable surface.

import { expect, test } from "@playwright/test";
import { loadLoginFixture } from "../support/load-login";

test("the product exposes its intended login contract", async ({ page }) => {
  await loadLoginFixture(page);

  await expect(page.getByLabel("Email")).toBeVisible();
  await expect(page.getByLabel("Password")).toBeVisible();
  await expect(page.getByRole("button", { name: "Sign in" })).toBeVisible();
});

This oracle does not duplicate every target-test step. It protects the disputed behavior: the product's accessible sign-in contract. The candidate must satisfy both the user journey and this independent acceptance fact.

Evidence Hashes and Manifests

A green console line is not enough. Each claim should point to preserved output with a hash, and each attempt should point to its complete result set.

{
  "schemaVersion": 1,
  "attempt": "02",
  "contractSha256": "<contract-hash>",
  "passed": true,
  "results": [
    { "id": "policy", "exitCode": 0, "passed": true, "logSha256": "<hash>" },
    { "id": "target-test", "exitCode": 0, "passed": true, "logSha256": "<hash>" },
    { "id": "immutable-oracle", "exitCode": 0, "passed": true, "logSha256": "<hash>" }
  ]
}

Evidence chain

• Check log hash proves which output the manifest describes.
• Attempt-manifest hash proves which result set the index describes.
• Contract hash proves which acceptance rules controlled the run.
• State file links the decision to the exact attempt manifest.

Independent Review and Human Gate

After checks pass, a separate reviewer challenges the result. Automation may recommend acceptance; a person owns the final risk decision.

Reviewer questions

1. Did every required check pass with matching evidence hashes?
2. Did a genuine failing attempt precede the candidate?
3. Did any protected file, assertion, credential, or test count change?
4. Does the correction address the observed cause without hiding risk?
5. Is the evidence complete enough for a human to reproduce the claim?

if (allChecksPass && hashesMatch && policyIntact && redBeforeGreen) {
  state = "AWAITING_HUMAN_APPROVAL";
} else {
  state = "ESCALATED";
}

The Synthetic Login Failure

The product correctly exposes a button named "Sign in." The target test is stale and asks for "Login." The exercise repairs the test without touching product behavior.

test("valid user reaches the dashboard", async ({ page }) => {
  await loadLoginFixture(page);
  await page.getByLabel("Email").fill("[email protected]");
  await page.getByLabel("Password").fill("correct-password");
  await page.getByRole("button", { name: "Login" }).click();

  await expect(page.getByRole("heading", { name: "Dashboard" })).toBeVisible();
});

Seal the Baseline

Setup restores the intentional failure, clears generated evidence, and hashes every protected tutorial file before the loop can act.

cd examples/qa-loop-demo
npm install
npx playwright install chromium
npm run setup

The expected setup result reports that the intentional failing test was restored, generated evidence was cleared, and the immutable file set was sealed. Read loop/baseline.json before proceeding.

Prove Red Before Green

Run the protected oracle separately, then run the target baseline. This distinguishes a stale test from a broken product and proves the exercise begins in the intended state.

npm run oracle
# 1 passed

npm run baseline
# expected failure:
# waiting for getByRole('button', { name: 'Login' })

If the oracle fails, stop: the product fixture no longer satisfies its own contract. If the target fails for another reason, stop: the exercise evidence does not support the planned correction.

Diagnose and Repair Narrowly

The failure names the missing accessible role and name. The oracle proves the intended name is "Sign in." The candidate changes only that stale locator.

- await page.getByRole("button", { name: "Login" }).click();
+ await page.getByRole("button", { name: "Sign in" }).click();

What remains unchanged

• The email and password inputs.
• The synthetic credentials.
• The dashboard assertion and test count.
• The product fixture, oracle, policy, controller, and verifier.

Verify Repeatedly

Run the controller. It records a failed attempt, applies the bounded correction, then requires policy, repeated target checks, and the immutable oracle to pass.

npm run loop

[verify:01] policy: PASS
[verify:01] target-test: FAIL
[verify:01] immutable-oracle: PASS
[verify:02] policy: PASS
[verify:02] target-test: PASS
[verify:02] immutable-oracle: PASS
[loop] AWAITING_HUMAN_APPROVAL

Inspect loop/LOOP_STATE.md, each attempt manifest, each command log, the evidence index, and the candidate file. Recompute hashes before trusting a pass claim.

Adapt the Pattern

Keep the same control shape while changing the artifact, verifier, oracle, and human owner for each QA domain.

Adaptation checklist

1. Replace the goal with a domain-observable outcome.
2. Keep one narrow mutable artifact.
3. Protect source requirements, datasets, policies, and acceptance oracles.
4. Require structured results and evidence hashes.
5. Escalate ambiguity, repeated failure, scope violation, and high-impact decisions.