DeFlaky lessons for QA.

Flake Taxonomy and Why Retries Hide Information

A flaky test produces inconsistent outcomes under conditions intended to be equivalent. A consistently failing test and a broken environment are different signals.

Classify before repair. Product races, test synchronization defects, unstable data, environment capacity, external dependencies, and requirement ambiguity can all produce intermittent outcomes. Repeating the same command reveals inconsistency; it does not identify the cause by itself.

A framework retry can make a CI job green while preserving the unstable first attempt. Keep retry metadata visible and establish a no-retry measurement lane when possible. Quarantine may protect delivery temporarily, but it must have an owner, defect, scope, and expiry or it becomes silent coverage loss.

Install and Run the Current CLI Safely

The reviewed CLI package is deflaky-cli 1.1.2 and exposes the deflaky binary with run and serve commands.

Pin the package version in CI or record the resolved version in evidence. A global installation exposes deflaky; no-install use can invoke npx deflaky-cli. The current run command requires --command, defaults to five runs and auto format, and accepts push, token, fail-threshold, and verbose options.

Run against a focused synthetic suite first. The wrapped command executes repeatedly with a five-minute timeout per run in the current source. It inherits the process environment, so strip production credentials and avoid commands that mutate shared systems.

npm install --global [email protected]
deflaky --version
deflaky --help

# Or without a global installation:
npx [email protected] run \
  --command "npx playwright test tests/flaky-demo.spec.ts" \
  --runs 5 \
  --format json \
  --verbose

Repeat a Command and Produce Parseable Reports

DeFlaky discovers report files after each run. The wrapped framework must write supported JUnit XML or JSON to a directory the runner scans.

Current search paths include test-results, reports, common Java build report paths, coverage, results, output, and report-like files in the working directory. Console-only output is not parsed into test cases.

Configure one deterministic report file that is overwritten by each run, or clear the dedicated exercise report directory as part of controlled setup. The current runner scans all matching files after every iteration, so stale or unrelated reports can duplicate identities and distort counts.

import { defineConfig } from "@playwright/test";

export default defineConfig({
  reporter: [
    ["line"],
    ["json", { outputFile: "test-results/deflaky-results.json" }],
  ],
  retries: 0,
  workers: 1,
});

rm -rf test-results
npx [email protected] run \
  -c "npx playwright test tests/flaky-demo.spec.ts --project=chromium" \
  -r 10 \
  --format json \
  --verbose

test -s test-results/deflaky-results.json
test -s .deflaky/last-run.json

Read the Current Classifier and FlakeScore

The shipped analyzer groups by test name, ignores skipped-only groups, classifies by pass rate, and scores only 100-percent-pass groups as stable.

For each test-name group, skipped results are removed. A 100 percent pass rate increments stableTests. A zero percent pass rate increments failingTests. Anything between zero and 100 increments flakyTests. The score is stableTests / totalTests * 100.

This implementation matters: a deterministically failing test lowers the score even though some README wording describes consistent pass or fail as stable. When zero tests are parsed, the analyzer returns 100 and the reporter prints a warning. Build your gate around both the score and the data-validity checks.

if (passRate === 100) stableTests++;
else if (passRate === 0) failingTests++;
else flakyTests++;

const totalTests = stableTests + flakyTests + failingTests;
const flakeScore = totalTests > 0
  ? (stableTests / totalTests) * 100
  : 100;

Reporter Contracts and Unique Test Identity

A pass-rate history is valid only when each observation belongs to the same logical test and each run contributes the intended report once.

The current analyzer uses tc.name as the map key. File path is stored but not included in identity. Duplicate titles in different files can therefore merge. Playwright JSON parsing prefixes a project name for nested specs, but JUnit and other JSON shapes may not provide equivalent disambiguation.

Make test titles globally unique within the measured corpus or add stable IDs to titles. Keep one report file for the focused command and validate that every test has exactly the expected number of non-skipped observations. Remove stale reports before measurement.

test("AUTH-LOCK-01 | API | locks fifth invalid attempt", async ({ request }) => {
  // ...
});

test("AUTH-LOCK-01 | E2E | blocks valid password while locked", async ({ page }) => {
  // ...
});

Separate Test, Product, Data, and Environment Causes

DeFlaky identifies outcome inconsistency. Root-cause analysis needs traces, logs, timings, state, infrastructure, and requirement evidence.

Choose one flaky test and preserve all passing and failing observations. Compare the earliest divergence, not only the final assertion. For UI tests inspect actionability and network timing. For API tests compare data and dependency responses. For distributed systems align test time with service and infrastructure metrics.

Do not jump from timeout to 'increase timeout.' Determine which readiness signal the test should observe. A product race may need a product fix; a missing web-first assertion may need a test fix; an overloaded runner may need environment capacity. Requirement ambiguity returns to the owner rather than code.

testId: "AUTH-LOCK-01 | E2E | blocks valid password while locked"
observed: "3 failures in 10 equivalent runs"
earliestDivergence: "audit event arrives after UI fetch on failed runs"
class: "product race"
evidence:
  - "trace-run-03.zip"
  - "auth-service-run-03.log"
  - "timeline-run-03.json"
repairOwner: "authentication team"
testOwner: "SDET"
reproductionCommand: "<same command used by DeFlaky>"

Inspect last-run.json and the Local Dashboard

Every run saves a machine-readable summary to .deflaky/last-run.json; the serve command renders that last run locally on port 3333 by default.

Inspect the JSON before the visual dashboard. Confirm command, timestamp, run count, total duration, total tests, category counts, score, and each detail history. Add .deflaky/ to ignore rules when reports should not be committed, or copy a sanitized evidence snapshot into a controlled artifact directory.

Run deflaky serve without a token for the local dashboard. Passing a token enables a request for AI analysis of flaky-test summaries to the DeFlaky API in the current server code. Review privacy, data classification, and token handling before enabling it.

node -e '
const d = require("./.deflaky/last-run.json");
if (d.result.totalTests < 1) throw new Error("No parsed tests");
console.log({
  command: d.command,
  runs: d.runs,
  score: d.result.flakeScore,
  flaky: d.result.flakyTests,
  failing: d.result.failingTests
});'

deflaky serve --port 3333

Dashboard review questions

• Does every detail show the expected number of observations?
• Do test names map to exactly one source identity?
• Are deterministic failures visible as failures, not called flaky?
• Does the saved command reproduce the same report paths?
• Is any uploaded or AI-analyzed data approved for that service?

Push Results and Configure a CI Threshold Securely

The current CLI can push parsed results with DEFLAKY_TOKEN and fail when FlakeScore is below a threshold, but functional correctness and data validity need separate gates.

Use a repository secret named DEFLAKY_TOKEN instead of placing a token in the command. The current source requires a token only when --push is set. A push failure is logged but does not by itself set a failing exit code; monitor upload success independently when dashboard history is required.

The threshold compares the implemented score after analysis. Pair it with the normal test command result, a non-empty parsed report check, expected unique test count, and zero deterministic failures for a merge gate. Validate the threshold is between zero and 100 in your workflow because the current CLI does not enforce that range.

- name: Functional test gate
  run: npx playwright test tests/auth --project=chromium

- name: Repeatability measurement
  run: |
    rm -rf test-results .deflaky
    npx [email protected] run \
      -c "npx playwright test tests/auth --project=chromium" \
      -r 5 --format json --fail-threshold 95 --push
    node scripts/validate-deflaky-result.mjs
  env:
    DEFLAKY_TOKEN: ${{ secrets.DEFLAKY_TOKEN }}

Playwright, Cypress, Jest, Mocha, Pytest, and JUnit Recipes

Framework support depends on report shape and file placement, not only on the command name. Prove one parsed fixture before scaling repetitions.

The current JSON parser handles Jest testResults, Mocha tests, a flat results array, and nested Playwright suites. The XML parser handles JUnit-style suites and cases. Pytest, Selenium, Java, and other frameworks work through JUnit XML when their reporter writes a discoverable file.

Configure retries to zero in the measurement lane and keep one report output. Run one iteration with --verbose, inspect discovered files, and compare expected names and statuses. Only then increase run count.

# 1. Clear only the dedicated measurement outputs.
rm -rf test-results .deflaky

# 2. Run once and inspect discovery.
npx [email protected] run -c "<focused framework command>" -r 1 --verbose

# 3. Validate totalTests, detail names, file paths, and status.
node scripts/validate-deflaky-result.mjs

# 4. Repeat only after the report contract passes.
npx [email protected] run -c "<same command>" -r 10

Establish a Baseline and Ratchet the Threshold

A useful baseline fixes command, environment, data, retries, workers, report contract, run count, and code revision so future scores are comparable.

Start with a focused stable corpus and at least enough repetitions to expose the timing window you care about. Record hardware or CI runner class, dependency services, browser, worker count, seed, and duration. Compare trends only when the experiment remains meaningfully equivalent.

Set an initial threshold below or equal to the validated baseline, then ratchet upward as named flakes are repaired. Changes to suite composition can move the score without changing any existing test's behavior, so retain per-test histories and deterministic failure counts alongside the aggregate.

{
  "commit": "<git-sha>",
  "command": "npx playwright test tests/auth --project=chromium",
  "runs": 10,
  "workers": 1,
  "retries": 0,
  "reporter": "playwright-json",
  "expectedTests": 12,
  "minimumScore": 92,
  "functionalGate": "required",
  "noReportGate": "required"
}

Audit Current Limitations and Docs/Source Mismatches

A production integration should encode the current implementation's edge cases so a misleading score cannot silently pass CI.

The reviewed 1.1.2 source and some public examples do not fully agree. The source exposes --fail-threshold and DEFLAKY_TOKEN; examples using other flag names or a standalone push/config command should be verified before use. The package name for no-install execution is deflaky-cli.

The source audit also reveals behaviors that need external controls: grouping by name only, scanning all matching report files every run, returning score 100 for zero parsed tests, counting deterministic failures against the score, ignoring command exit codes in the analyzer, and treating push failure as nonfatal.

import data from "../.deflaky/last-run.json" with { type: "json" };

const { result, runs } = data;
if (result.totalTests < 1) throw new Error("DeFlaky parsed no tests");
if (result.totalTests !== Number(process.env.EXPECTED_TESTS)) {
  throw new Error("Expected " + process.env.EXPECTED_TESTS + ", got " + result.totalTests);
}
if (result.failingTests > 0) throw new Error("Deterministic failures remain");
for (const test of result.details) {
  if (test.totalRuns !== runs) throw new Error("Incomplete history: " + test.testName);
}

Capstone: Reproduce, Classify, Repair, and Prove Stability

Create one controlled timing flake, measure it, preserve a failing counterexample, repair the synchronization defect, and rerun the identical experiment.

The synthetic page starts a status update after a variable but seeded delay. The original test clicks Start and reads the status immediately. Some runs observe Pending and others Complete. The repair waits for the observable Complete state with a web-first assertion; it does not add a fixed sleep or global retry.

Use a unique test title and JSON reporter. Run ten iterations before and after under the same worker, retry, seed, and environment settings. Keep the normal functional gate and DeFlaky validity script. A reviewer should be able to reproduce both the original mixed history and the accepted stable history.

// Before: races the product update
await page.getByRole("button", { name: "Start" }).click();
expect(await page.getByTestId("status").textContent()).toBe("Complete");

// After: observes the user-visible readiness condition
await page.getByRole("button", { name: "Start" }).click();
await expect(page.getByTestId("status")).toHaveText("Complete");

rm -rf test-results .deflaky
npx [email protected] run \
  -c "npx playwright test tests/flaky-demo.spec.ts --project=chromium" \
  -r 10 --format json --verbose
node scripts/validate-deflaky-result.mjs

# Archive .deflaky/last-run.json and raw report as BEFORE or AFTER.
# The command, runs, project, retries, workers, and seed stay unchanged.

Acceptance package

1. Pinned CLI and framework versions plus reporter configuration.
2. Original mixed history with trace from at least one failed run.
3. Root-cause record naming the missing observable wait.
4. One narrow diff with no sleep, forced action, quarantine, or retry increase.
5. After history with expected count, complete observations, no failures, and score 100.
6. Independent reproduction and a CI threshold that keeps all validity gates.