--- name: flaky-test-detector description: >- Find flaky tests by running a suite several times and comparing the results, then score each test, print a table in the terminal, and write an HTML report. Works with Playwright (JSON reporter) and Selenium / TestNG / JUnit (XML reports), detected automatically. Use when a tester or SDET says "find the flaky tests", "why does CI fail randomly", "is this test flaky", "run the suite 5 times and tell me what is unstable", "which tests should we quarantine", or "give me a flakiness report". Returns a per-test flakiness rate, a suite FlakeScore out of 100, and a quarantine shortlist, always as a draft for a human to confirm. license: MIT metadata: author: TheTestingAcademy stlc-phase: Test Execution version: 1.0.0 --- # Flaky Test Detector Owns: proving which tests change their mind on unchanged code, and by how much. Must not: "fix" flakiness by adding retries or bumping timeouts, and must not quarantine anything on its own. ## When to use - CI goes red, someone hits re-run, and it goes green. Nobody knows which test did it. - Before a release, to know whether a failure is a real bug or noise. - A tester asks "is this test flaky or actually broken?" (those need opposite responses). - You need a shortlist of what to quarantine, with numbers behind it. ## Workflow 1. **Gather inputs.** Ask for (or infer) the test command and how many runs. Defaults: 5 runs to smell it, 10 to trust it. Never invent results, never simulate a run: every number must come from a real execution. - Playwright needs the JSON reporter: `npx playwright test --reporter=json` - Selenium / TestNG / JUnit just needs its usual XML reports on disk (surefire-reports, target, test-results). 2. **Collect.** Run the suite N times and record every attempt: ``` python scripts/collect.py --command "npx playwright test --reporter=json" --runs 5 --fresh ``` Appends one line per attempt to `.flaky/history.jsonl`. Non-zero exits are expected: a failing test is the signal, not an error. Re-running without `--fresh` adds to the history, so flakiness sharpens over time. 3. **Score.** Compute the numbers and classify: ``` python scripts/analyze.py --html flaky-report.html --json flaky.json ``` The math, per test, over its chronological attempts: ``` flakiness rate = failures / attempts flip rate = outcome changes / (attempts - 1) FlakeScore = 100 x (1 - sum(flakiness rate of FLAKY tests) / total tests) ``` Classify into three buckets, and keep them apart: - **STABLE** - zero failures. - **FLAKY** - passed at least once AND failed at least once. Flagged at flakiness >= 10%. - **BROKEN** - never passed. This is NOT flake, it is a real failure. Say so and stop calling it flaky. Quarantine shortlist = FLAKY with flakiness rate >= 35%. 4. **Categorize the root cause** for each flaky test, from its failure message and timing. Name one category per test and say what evidence points there: | Category | Typical signal | The fix that actually works | |---|---|---| | timing | timeout waiting for element/text | explicit condition wait, not a sleep | | race-condition | click before enabled, nav vs DOM | wait on the response/state, not the clock | | state-leakage | fails only after another test | cleanup in before/after hooks | | external-dependency | network/API/3rd-party error | mock or stub the call | | resource-contention | grid/session/port errors | isolate or serialize the resource | | test-data | duplicate/stale record | generate unique data per run | | environment-specific | one browser/OS/CI only | pin or fix the environment | | unknown | nothing conclusive | say unknown, ask for a trace or video | If the evidence does not support a category, the category is `unknown`. Never guess a root cause to fill the column. 5. **HUMAN REVIEW GATE (mandatory).** Present the table + report as a DRAFT. State: how many runs it is based on (5 runs is a smell, not proof), which verdicts are low-confidence, what you assumed, and that BROKEN tests are real failures needing a fix, not quarantine. Ask the tester to confirm the quarantine shortlist before anything is skipped, tagged, or filed. Do not quarantine, edit tests, add retries, or open tickets on your own. ## Output shape ``` Flaky Test Report 10 runs - playwright, selenium/junit - 8 tests ------------------------------------------------------------------------------------- TEST RESULTS PASS FAIL RATE FLIP VERDICT ------------------------------------------------------------------------------------- checkout.spec.ts > pay button enables after .. x/x/x/xx/x 4 6 60% 89% FLAKY +QUARANTINE CheckoutTest.applyCouponUpdatesTotal /xx///x/// 7 3 30% 44% FLAKY search.spec.ts > search returns results xxxxxxxxxx 0 10 100% 0% BROKEN LoginTest.loginWithValidUser ////////// 10 0 0% 0% STABLE ------------------------------------------------------------------------------------- FlakeScore: 83.8 / 100 2 stable 5 flaky 1 broken 1 to quarantine HTML report: flaky-report.html --- HUMAN REVIEW GATE --- Based on: 10 runs, same commit, one machine. Root cause (advisory): pay button = race-condition (clicks before enabled). GridTest = resource-contention (session could not start). Low confidence: cart badge (10%, 1 failure in 10) could be noise, needs more runs. Note: "search returns results" is BROKEN, not flaky - 0/10 passes. Fix it, do not quarantine it. Confirm the quarantine shortlist (1 test) before I tag anything. ``` ## Guardrails - Never fabricate or simulate a run. Every number traces to a real execution recorded in `.flaky/history.jsonl`. - Never "fix" flake with retries, `sleep`, or bigger timeouts. That hides the signal. Report the root cause instead. - BROKEN is not FLAKY. A test that never passes is a real failure; calling it flake buries a bug. - Small samples lie. Below 5 runs, say so out loud; a 1-in-10 failure may be noise. More runs, more truth. - Quarantine is a human decision, and quarantine is not deletion: the test stays in the codebase and keeps running in a separate non-blocking pipeline until it is fixed. - Root cause is advisory. If the evidence is thin, the answer is `unknown` plus what you would need (trace, video, CI logs). - Stay in this lane: detect and measure. Fixing the test, filing the ticket, and changing CI belong to a human and to other skills.