Build a Flaky Test AI Agent

Everyone knows the ritual: CI goes red, someone hits re-run, it goes green, the PR merges. Nobody writes down which test did it. Do that for six months and the suite becomes a coin toss nobody trusts. This guide builds an agent skill that ends the guessing: it runs the suite N times, records every attempt, and tells you exactly which tests are flaky, how flaky, and which are not flaky at all but simply broken.

1. Flaky vs broken

A test is flaky when it produces different results on the same code. That last part matters: if the code changed, a new failure is just a failure. Flakiness only means anything when the input is held still and the output still moves.

Run the suite five times and every test lands in one of three buckets. Most teams only have two words for this, which is exactly why flaky tests survive so long.

2. How the agent works

You ask once. The skill runs your suite N times, captures whatever your framework already writes (Playwright's JSON reporter, or the JUnit/TestNG XML that Selenium and Maven already produce), appends every attempt to a history file, then scores it. Nothing about your tests changes, and no plugin gets installed.

3. The formula

Getting this right is the difference between a report people act on and a number people argue about. Per test, over its attempts in chronological order:

And the classifier, which is deliberately blunt:

4. The prompt that builds it

You do not have to hand-write any of this. Paste the prompt below into GitHub Copilot or Claude Code and it will generate the skill folder for you. It is long on purpose: the specifics (three buckets, the exact formula, the review gate) are what stop the model from producing a vague retry-wrapper instead of a detector.

Create an agent skill called flaky-test-detector.

Goal: find flaky tests by running a test suite several times on the SAME code and
comparing the results. Support Playwright (JSON reporter) and Selenium/TestNG/JUnit
(XML reports), and detect which one is in use automatically.

Build it as a folder: SKILL.md plus scripts/collect.py and scripts/analyze.py.
Python standard library only, no pip installs.

collect.py  --command "<test cmd>" --runs N [--fresh]
  Runs the command N times. A non-zero exit is expected (failing tests are the
  signal). After each run, parse results and append ONE JSON line per test attempt
  to .flaky/history.jsonl: {run, framework, id, file, status, duration_ms, message}.
  Playwright: parse the JSON reporter from stdout, walking suites -> specs -> tests
  -> results so retries count as separate attempts.
  Selenium: parse JUnit XML from surefire-reports/target/test-results, only files
  modified by this run.

analyze.py  [--html FILE] [--json FILE] [--detect 0.10] [--quarantine 0.35]
  Group attempts by test id, in chronological order, and compute:
    flakiness rate = failures / attempts
    flip rate      = outcome changes / (attempts - 1)
    FlakeScore     = 100 * (1 - sum(flakiness of FLAKY tests) / total tests)
  Classify into exactly three buckets and never mix them up:
    STABLE = zero failures
    FLAKY  = at least one pass AND at least one fail, flagged at rate >= 0.10
    BROKEN = never passed. This is a REAL failure, not flake. Say so.
  Mark FLAKY tests with rate >= 0.35 as quarantine candidates.
  Print a terminal table sorted flaky-first with a pass/fail bead strip per test,
  and write a self-contained HTML report with the same data plus the formula.

Rules for SKILL.md: never fabricate a run, never fix flake with retries or longer
timeouts, never quarantine on its own. It ends at a human review gate that states
the run count, flags low-confidence verdicts, and asks before anything is
quarantined. Suggest a root-cause category per flaky test (timing, race-condition,
state-leakage, external-dependency, resource-contention, test-data,
environment-specific, unknown) from the failure message, and say "unknown" when
the evidence is thin.

Prefer the finished article? Everything it generates is below, and downloadable at the bottom.

5. The full SKILL.md

The brain. The description is what makes the agent pick this skill on its own when you say "why does CI keep failing randomly".

---
name: flaky-test-detector
description: >-
  Find flaky tests by running a suite several times and comparing the results,
  then score each test, print a table in the terminal, and write an HTML report.
  Works with Playwright (JSON reporter) and Selenium / TestNG / JUnit (XML
  reports), detected automatically. Use when a tester or SDET says "find the
  flaky tests", "why does CI fail randomly", "is this test flaky", "run the
  suite 5 times and tell me what is unstable", "which tests should we
  quarantine", or "give me a flakiness report". Returns a per-test flakiness
  rate, a suite FlakeScore out of 100, and a quarantine shortlist, always as a
  draft for a human to confirm.
license: MIT
metadata:
  author: TheTestingAcademy
  stlc-phase: Test Execution
  version: 1.0.0
---

# Flaky Test Detector

Owns: proving which tests change their mind on unchanged code, and by how much. Must not: "fix" flakiness by adding retries or bumping timeouts, and must not quarantine anything on its own.

## When to use
- CI goes red, someone hits re-run, and it goes green. Nobody knows which test did it.
- Before a release, to know whether a failure is a real bug or noise.
- A tester asks "is this test flaky or actually broken?" (those need opposite responses).
- You need a shortlist of what to quarantine, with numbers behind it.

## Workflow

1. **Gather inputs.** Ask for (or infer) the test command and how many runs. Defaults: 5 runs to smell it, 10 to trust it. Never invent results, never simulate a run: every number must come from a real execution.
   - Playwright needs the JSON reporter: `npx playwright test --reporter=json`
   - Selenium / TestNG / JUnit just needs its usual XML reports on disk (surefire-reports, target, test-results).

2. **Collect.** Run the suite N times and record every attempt:
   ```
   python scripts/collect.py --command "npx playwright test --reporter=json" --runs 5 --fresh
   ```
   Appends one line per attempt to `.flaky/history.jsonl`. Non-zero exits are expected: a failing test is the signal, not an error. Re-running without `--fresh` adds to the history, so flakiness sharpens over time.

3. **Score.** Compute the numbers and classify:
   ```
   python scripts/analyze.py --html flaky-report.html --json flaky.json
   ```
   The math, per test, over its chronological attempts:
   ```
   flakiness rate = failures / attempts
   flip rate      = outcome changes / (attempts - 1)
   FlakeScore     = 100 x (1 - sum(flakiness rate of FLAKY tests) / total tests)
   ```
   Classify into three buckets, and keep them apart:
   - **STABLE** - zero failures.
   - **FLAKY** - passed at least once AND failed at least once. Flagged at flakiness >= 10%.
   - **BROKEN** - never passed. This is NOT flake, it is a real failure. Say so and stop calling it flaky.

   Quarantine shortlist = FLAKY with flakiness rate >= 35%.

4. **Categorize the root cause** for each flaky test, from its failure message and timing. Name one category per test and say what evidence points there:

   | Category | Typical signal | The fix that actually works |
   |---|---|---|
   | timing | timeout waiting for element/text | explicit condition wait, not a sleep |
   | race-condition | click before enabled, nav vs DOM | wait on the response/state, not the clock |
   | state-leakage | fails only after another test | cleanup in before/after hooks |
   | external-dependency | network/API/3rd-party error | mock or stub the call |
   | resource-contention | grid/session/port errors | isolate or serialize the resource |
   | test-data | duplicate/stale record | generate unique data per run |
   | environment-specific | one browser/OS/CI only | pin or fix the environment |
   | unknown | nothing conclusive | say unknown, ask for a trace or video |

   If the evidence does not support a category, the category is `unknown`. Never guess a root cause to fill the column.

5. **HUMAN REVIEW GATE (mandatory).** Present the table + report as a DRAFT. State: how many runs it is based on (5 runs is a smell, not proof), which verdicts are low-confidence, what you assumed, and that BROKEN tests are real failures needing a fix, not quarantine. Ask the tester to confirm the quarantine shortlist before anything is skipped, tagged, or filed. Do not quarantine, edit tests, add retries, or open tickets on your own.

## Output shape

```
Flaky Test Report  10 runs - playwright, selenium/junit - 8 tests
-------------------------------------------------------------------------------------
TEST                                            RESULTS     PASS FAIL  RATE  FLIP  VERDICT
-------------------------------------------------------------------------------------
checkout.spec.ts > pay button enables after ..  x/x/x/xx/x     4    6   60%   89%  FLAKY +QUARANTINE
CheckoutTest.applyCouponUpdatesTotal            /xx///x///     7    3   30%   44%  FLAKY
search.spec.ts > search returns results         xxxxxxxxxx     0   10  100%    0%  BROKEN
LoginTest.loginWithValidUser                    //////////    10    0    0%    0%  STABLE
-------------------------------------------------------------------------------------
FlakeScore: 83.8 / 100   2 stable  5 flaky  1 broken  1 to quarantine

HTML report: flaky-report.html

--- HUMAN REVIEW GATE ---
Based on: 10 runs, same commit, one machine.
Root cause (advisory): pay button = race-condition (clicks before enabled).
                       GridTest  = resource-contention (session could not start).
Low confidence: cart badge (10%, 1 failure in 10) could be noise, needs more runs.
Note: "search returns results" is BROKEN, not flaky - 0/10 passes. Fix it, do not quarantine it.
Confirm the quarantine shortlist (1 test) before I tag anything.
```

## Guardrails
- Never fabricate or simulate a run. Every number traces to a real execution recorded in `.flaky/history.jsonl`.
- Never "fix" flake with retries, `sleep`, or bigger timeouts. That hides the signal. Report the root cause instead.
- BROKEN is not FLAKY. A test that never passes is a real failure; calling it flake buries a bug.
- Small samples lie. Below 5 runs, say so out loud; a 1-in-10 failure may be noise. More runs, more truth.
- Quarantine is a human decision, and quarantine is not deletion: the test stays in the codebase and keeps running in a separate non-blocking pipeline until it is fixed.
- Root cause is advisory. If the evidence is thin, the answer is `unknown` plus what you would need (trace, video, CI logs).
- Stay in this lane: detect and measure. Fixing the test, filing the ticket, and changing CI belong to a human and to other skills.

6. Script: collect.py

Runs the suite N times and records every attempt. The only clever bits: a non-zero exit code is expected (a failing test is the data, not an error), retries count as separate attempts, and JUnit XML is filtered by modification time so last week's reports do not leak into today's numbers.

#!/usr/bin/env python3
"""Run a test suite N times and record every attempt to a history file.

Detects the result format on its own:
  - Playwright : JSON reporter output  (--reporter=json)
  - Selenium / JUnit / TestNG : JUnit XML (surefire-reports, target, test-results)

Usage:
  python collect.py --command "npx playwright test --reporter=json" --runs 5
  python collect.py --command "mvn -q test" --runs 5 --format junit
  python collect.py --command "npx playwright test --reporter=json" --runs 5 --fresh

Writes .flaky/history.jsonl, one JSON object per test attempt. Then run analyze.py.
Stdlib only. No install step.
"""
import argparse, json, os, subprocess, sys, glob, time, datetime
import xml.etree.ElementTree as ET


# ---------- playwright ----------
def parse_playwright(stdout, run):
    """Walk the Playwright JSON reporter tree. Every result (incl. retries) is an attempt."""
    start = stdout.find("{")
    if start == -1:
        return []
    try:
        data = json.loads(stdout[start:])
    except json.JSONDecodeError:
        return []
    rows = []

    def walk(suite, path):
        title = suite.get("title", "")
        here = path + [title] if title else path
        for spec in suite.get("specs", []) or []:
            file = spec.get("file") or suite.get("file") or ""
            # Playwright repeats the file as the top suite title. Drop it so the id
            # reads "tests/checkout.spec.ts > describe > test", not the name twice.
            base = os.path.basename(file)
            crumbs = [p for p in here if p and p != file and p != base]
            name = " > ".join(crumbs + [spec.get("title", "")])
            tid = f"{file} > {name}" if file else name
            for test in spec.get("tests", []) or []:
                for res in test.get("results", []) or []:
                    st = res.get("status")
                    st = {"passed": "passed", "failed": "failed", "timedOut": "failed",
                          "interrupted": "failed", "skipped": "skipped"}.get(st, st)
                    msg = ""
                    for err_key in ("error", "errors"):
                        err = res.get(err_key)
                        if isinstance(err, dict):
                            msg = (err.get("message") or "").strip()
                        elif isinstance(err, list) and err:
                            msg = (err[0].get("message") or "").strip()
                        if msg:
                            break
                    rows.append({"run": run, "framework": "playwright", "id": tid, "file": file,
                                 "name": spec.get("title", ""), "status": st,
                                 "duration_ms": res.get("duration"),
                                 "message": msg.splitlines()[0][:300] if msg else ""})
        for child in suite.get("suites", []) or []:
            walk(child, here)

    for s in data.get("suites", []) or []:
        walk(s, [])
    return rows


# ---------- junit / testng (selenium, maven surefire) ----------
JUNIT_GLOBS = ["**/surefire-reports/*.xml", "**/failsafe-reports/*.xml",
               "**/target/**/TEST-*.xml", "**/test-results/**/*.xml",
               "**/build/test-results/**/*.xml", "**/reports/**/TEST-*.xml"]


def find_junit(since):
    seen, files = set(), []
    for g in JUNIT_GLOBS:
        for f in glob.glob(g, recursive=True):
            if f in seen or not os.path.isfile(f):
                continue
            # only files this run touched, so old reports do not leak in
            if since and os.path.getmtime(f) < since - 1:
                continue
            seen.add(f)
            files.append(f)
    return files


def parse_junit(files, run):
    rows = []
    for f in files:
        try:
            root = ET.parse(f).getroot()
        except ET.ParseError:
            continue
        suites = [root] if root.tag == "testsuite" else root.iter("testsuite")
        for suite in suites:
            for tc in suite.iter("testcase"):
                cls = tc.get("classname") or suite.get("name") or ""
                name = tc.get("name") or ""
                tid = f"{cls}.{name}" if cls else name
                status, msg = "passed", ""
                if tc.find("skipped") is not None:
                    status = "skipped"
                for tag in ("failure", "error"):
                    node = tc.find(tag)
                    if node is not None:
                        status = "failed"
                        msg = (node.get("message") or (node.text or "")).strip()
                        break
                dur = tc.get("time")
                rows.append({"run": run, "framework": "selenium/junit", "id": tid, "file": cls,
                             "name": name, "status": status,
                             "duration_ms": int(float(dur) * 1000) if dur else None,
                             "message": msg.splitlines()[0][:300] if msg else ""})
    return rows


def main():
    ap = argparse.ArgumentParser(description="Run a suite N times and record every attempt.")
    ap.add_argument("--command", required=True, help='e.g. "npx playwright test --reporter=json"')
    ap.add_argument("--runs", type=int, default=5)
    ap.add_argument("--format", choices=["auto", "playwright", "junit"], default="auto")
    ap.add_argument("--history", default=".flaky/history.jsonl")
    ap.add_argument("--fresh", action="store_true", help="wipe history first (new baseline)")
    a = ap.parse_args()

    os.makedirs(os.path.dirname(a.history) or ".", exist_ok=True)
    if a.fresh and os.path.exists(a.history):
        os.remove(a.history)

    start_run = 1
    if os.path.exists(a.history):
        with open(a.history) as fh:
            runs = [json.loads(l).get("run", 0) for l in fh if l.strip()]
        start_run = (max(runs) + 1) if runs else 1

    total = 0
    for i in range(a.runs):
        run = start_run + i
        t0 = time.time()
        print(f"[{run}] {a.command}", flush=True)
        proc = subprocess.run(a.command, shell=True, capture_output=True, text=True)
        # A non-zero exit is EXPECTED here: failing tests are the signal, not an error.

        rows = []
        if a.format in ("auto", "playwright"):
            rows = parse_playwright(proc.stdout, run)
        if not rows and a.format in ("auto", "junit"):
            rows = parse_junit(find_junit(t0), run)

        if not rows:
            print(f"    no results parsed. Playwright needs --reporter=json; "
                  f"JUnit/TestNG needs XML reports on disk.", file=sys.stderr)
        else:
            with open(a.history, "a") as fh:
                for r in rows:
                    r["ts"] = datetime.datetime.now().isoformat(timespec="seconds")
                    fh.write(json.dumps(r) + "\n")
            p = sum(1 for r in rows if r["status"] == "passed")
            f = sum(1 for r in rows if r["status"] == "failed")
            total += len(rows)
            print(f"    {len(rows)} attempts  {p} passed  {f} failed  ({time.time()-t0:.1f}s)")

    print(f"\n{total} attempts appended to {a.history}")
    print("Next: python analyze.py --html flaky-report.html")


if __name__ == "__main__":
    main()

7. Script: analyze.py

The math, the terminal table, and the HTML report. Pure standard library, so there is nothing to pip install and nothing to keep up to date.

#!/usr/bin/env python3
"""Score every test in the run history, print a CLI table, write an HTML report.

Reads the JSONL history written by collect.py (one line per test attempt) and
answers one question: which tests changed their mind on unchanged code?

Usage:
  python analyze.py                                   # table only
  python analyze.py --html flaky-report.html          # table + HTML report
  python analyze.py --json flaky.json                 # machine-readable
  python analyze.py --detect 0.10 --quarantine 0.35   # tune thresholds

Stdlib only. No install step.
"""
import argparse, json, os, sys, html, datetime, collections

DETECT_DEFAULT = 0.10      # flakiness rate at/above this = FLAKY
QUARANTINE_DEFAULT = 0.35  # flakiness rate at/above this = quarantine candidate


# ---------- math ----------
def score_test(attempts):
    """attempts = chronological list of 'passed'/'failed'. Returns the stats dict.

    flakiness_rate = failures / attempts          (the headline number)
    flip_rate      = flips / (attempts - 1)       (how erratic the order is)
    A test that never passes is BROKEN, not flaky: it is a real failure.
    """
    outcomes = [a for a in attempts if a in ("passed", "failed")]
    total = len(outcomes)
    if total == 0:
        return None
    passes = outcomes.count("passed")
    failures = outcomes.count("failed")
    flips = sum(1 for a, b in zip(outcomes, outcomes[1:]) if a != b)

    flakiness_rate = failures / total
    pass_rate = passes / total
    flip_rate = flips / (total - 1) if total > 1 else 0.0

    if failures == 0:
        verdict = "STABLE"
    elif passes == 0:
        verdict = "BROKEN"          # always fails: fix it, do not call it flake
    else:
        verdict = "FLAKY"

    return {
        "attempts": total, "passes": passes, "failures": failures,
        "flips": flips, "flakiness_rate": flakiness_rate,
        "pass_rate": pass_rate, "flip_rate": flip_rate,
        "verdict": verdict, "outcomes": outcomes,
    }


def flake_score(tests):
    """Suite reliability, 0-100. 100 = nothing flaky.

    Only FLAKY tests drag the score down. BROKEN tests are failures, not flake,
    so they are reported separately instead of being averaged into the score.
    """
    if not tests:
        return 100.0
    penalty = sum(t["flakiness_rate"] for t in tests.values() if t["verdict"] == "FLAKY")
    return round(100.0 * (1 - penalty / len(tests)), 1)


# ---------- io ----------
def load(history):
    if not os.path.exists(history):
        sys.exit(f"no history at {history}. Run collect.py first.")
    by_test = collections.OrderedDict()
    meta = {"runs": set(), "frameworks": set()}
    with open(history) as fh:
        for line in fh:
            line = line.strip()
            if not line:
                continue
            try:
                r = json.loads(line)
            except json.JSONDecodeError:
                continue
            if r.get("status") == "skipped":
                continue
            by_test.setdefault(r["id"], {"id": r["id"], "file": r.get("file", ""),
                                         "framework": r.get("framework", ""),
                                         "attempts": [], "messages": []})
            by_test[r["id"]]["attempts"].append(r.get("status"))
            if r.get("status") == "failed" and r.get("message"):
                by_test[r["id"]]["messages"].append(r["message"])
            meta["runs"].add(r.get("run"))
            if r.get("framework"):
                meta["frameworks"].add(r["framework"])
    return by_test, meta


def analyze(history, detect, quarantine):
    raw, meta = load(history)
    tests = {}
    for tid, t in raw.items():
        s = score_test(t["attempts"])
        if not s:
            continue
        s.update({"id": tid, "file": t["file"], "framework": t["framework"],
                  "message": (t["messages"] or [""])[0][:300]})
        if s["verdict"] == "FLAKY" and s["flakiness_rate"] < detect:
            s["verdict"] = "STABLE"  # below the detection floor: treat as noise-free
        s["quarantine"] = s["verdict"] == "FLAKY" and s["flakiness_rate"] >= quarantine
        tests[tid] = s
    return tests, meta


# ---------- cli table ----------
def bar(t):
    return "".join("✓" if o == "passed" else "✗" for o in t["outcomes"])


def cli_table(tests, meta, score, detect, quarantine, color=True):
    C = {"g": "\033[32m", "r": "\033[31m", "y": "\033[33m", "b": "\033[1m", "d": "\033[2m", "x": "\033[0m"}
    if not color:
        C = {k: "" for k in C}

    order = {"FLAKY": 0, "BROKEN": 1, "STABLE": 2}
    rows = sorted(tests.values(), key=lambda t: (order[t["verdict"]], -t["flakiness_rate"], t["id"]))
    runs = len([r for r in meta["runs"] if r is not None])
    fw = ", ".join(sorted(meta["frameworks"])) or "unknown"

    name_w = min(max([len(shorten(t["id"])) for t in rows] + [16]), 54)
    hdr = f"{'TEST':<{name_w}}  {'RESULTS':<10} {'PASS':>5} {'FAIL':>5} {'RATE':>6} {'FLIP':>6}  VERDICT"
    line = "-" * len(hdr)

    out = []
    out.append("")
    out.append(f"{C['b']}Flaky Test Report{C['x']}  {C['d']}{runs} runs - {fw} - {len(tests)} tests{C['x']}")
    out.append(line)
    out.append(f"{C['b']}{hdr}{C['x']}")
    out.append(line)
    for t in rows:
        col = C["y"] if t["verdict"] == "FLAKY" else (C["r"] if t["verdict"] == "BROKEN" else C["g"])
        v = t["verdict"] + (" +QUARANTINE" if t.get("quarantine") else "")
        out.append(f"{shorten(t['id']):<{name_w}}  {bar(t):<10} {t['passes']:>5} {t['failures']:>5} "
                   f"{t['flakiness_rate']*100:>5.0f}% {t['flip_rate']*100:>5.0f}%  {col}{v}{C['x']}")
    out.append(line)

    n_flaky = sum(1 for t in tests.values() if t["verdict"] == "FLAKY")
    n_broken = sum(1 for t in tests.values() if t["verdict"] == "BROKEN")
    n_stable = sum(1 for t in tests.values() if t["verdict"] == "STABLE")
    n_q = sum(1 for t in tests.values() if t.get("quarantine"))
    sc = C["g"] if score >= 95 else (C["y"] if score >= 80 else C["r"])
    out.append(f"{C['b']}FlakeScore: {sc}{score} / 100{C['x']}   "
               f"{C['g']}{n_stable} stable{C['x']}  {C['y']}{n_flaky} flaky{C['x']}  "
               f"{C['r']}{n_broken} broken{C['x']}  {n_q} to quarantine")
    out.append(f"{C['d']}flakiness rate = failures / attempts   "
               f"detect >= {detect*100:.0f}%   quarantine >= {quarantine*100:.0f}%{C['x']}")
    out.append("")
    return "\n".join(out)


def shorten(tid, n=54):
    return tid if len(tid) <= n else "..." + tid[-(n - 3):]


# ---------- html report ----------
def html_report(tests, meta, score, detect, quarantine):
    e = lambda s: html.escape(str(s), quote=True)
    rows = sorted(tests.values(), key=lambda t: ({"FLAKY": 0, "BROKEN": 1, "STABLE": 2}[t["verdict"]],
                                                 -t["flakiness_rate"], t["id"]))
    runs = len([r for r in meta["runs"] if r is not None])
    fw = ", ".join(sorted(meta["frameworks"])) or "unknown"
    n = {v: sum(1 for t in tests.values() if t["verdict"] == v) for v in ("STABLE", "FLAKY", "BROKEN")}
    n_q = sum(1 for t in tests.values() if t.get("quarantine"))
    ring = "#16a34a" if score >= 95 else ("#ea580c" if score >= 80 else "#dc2626")
    ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M")

    trs = []
    for t in rows:
        cls = t["verdict"].lower()
        beads = "".join(
            f'<i class="{"p" if o == "passed" else "f"}"></i>' for o in t["outcomes"])
        q = '<span class="q">QUARANTINE</span>' if t.get("quarantine") else ""
        msg = f'<div class="msg">{e(t["message"])}</div>' if t["message"] else ""
        trs.append(f'''<tr class="{cls}">
      <td><div class="tid">{e(t["id"])}</div>{msg}</td>
      <td class="beads">{beads}</td>
      <td class="num">{t["passes"]}</td>
      <td class="num">{t["failures"]}</td>
      <td class="num"><b>{t["flakiness_rate"]*100:.0f}%</b></td>
      <td class="num">{t["flip_rate"]*100:.0f}%</td>
      <td><span class="v {cls}">{t["verdict"]}</span>{q}</td>
    </tr>''')

    return f'''<!DOCTYPE html>
<html lang="en"><head><meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>Flaky Test Report - {e(fw)}</title>
<style>
 :root{{--ink:#1a1c23;--sub:#5b6472;--line:#e4e7ec;--bg:#fbfaf7;--card:#fff;--accent:#7c3aed;
   --ok:#16a34a;--warn:#ea580c;--bad:#dc2626}}
 @media(prefers-color-scheme:dark){{:root{{--ink:#e9edf4;--sub:#9aa6b8;--line:#2a2f3a;--bg:#0d0f14;--card:#151922}}}}
 *{{box-sizing:border-box}}
 body{{margin:0;background:var(--bg);color:var(--ink);font:15px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,sans-serif;padding:32px 20px 60px}}
 .wrap{{max-width:1000px;margin:0 auto}}
 h1{{font-size:26px;margin:0 0 4px;letter-spacing:-.02em}}
 .meta{{color:var(--sub);font-size:14px;margin-bottom:22px}}
 .cards{{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px;margin-bottom:26px}}
 .c{{background:var(--card);border:1px solid var(--line);border-radius:12px;padding:16px}}
 .c .k{{color:var(--sub);font-size:12px;text-transform:uppercase;letter-spacing:.08em;font-weight:600}}
 .c .val{{font-size:28px;font-weight:800;margin-top:4px}}
 .score .val{{color:{ring}}}
 table{{width:100%;border-collapse:collapse;background:var(--card);border:1px solid var(--line);border-radius:12px;overflow:hidden}}
 th{{text-align:left;font-size:11px;text-transform:uppercase;letter-spacing:.07em;color:var(--sub);padding:11px 12px;border-bottom:1px solid var(--line)}}
 td{{padding:11px 12px;border-bottom:1px solid var(--line);vertical-align:top}}
 tr:last-child td{{border-bottom:none}}
 .num{{text-align:right;font-variant-numeric:tabular-nums;white-space:nowrap}}
 .tid{{font:500 13px ui-monospace,SFMono-Regular,Menlo,monospace;word-break:break-all}}
 .msg{{color:var(--bad);font:400 11.5px ui-monospace,monospace;margin-top:5px;opacity:.85}}
 .beads{{white-space:nowrap}}
 .beads i{{display:inline-block;width:9px;height:9px;border-radius:50%;margin-right:3px}}
 .beads .p{{background:var(--ok)}} .beads .f{{background:var(--bad)}}
 .v{{font-size:11px;font-weight:800;letter-spacing:.05em;padding:3px 8px;border-radius:6px}}
 .v.flaky{{background:#fff2e0;color:#b45309}} .v.broken{{background:#fee;color:var(--bad)}}
 .v.stable{{background:#e8f8ee;color:var(--ok)}}
 @media(prefers-color-scheme:dark){{.v.flaky{{background:#4a2f0a}}.v.broken{{background:#4a1414}}.v.stable{{background:#0e3a20}}}}
 .q{{margin-left:6px;font-size:10px;font-weight:800;color:#fff;background:var(--bad);padding:3px 7px;border-radius:6px}}
 .formula{{margin-top:26px;background:var(--card);border:1px solid var(--line);border-left:4px solid var(--accent);border-radius:10px;padding:14px 18px;font-size:13.5px;color:var(--sub)}}
 .formula code{{color:var(--accent);font-weight:600}}
 footer{{margin-top:28px;color:var(--sub);font-size:13px}}
</style></head><body><div class="wrap">
<h1>Flaky Test Report</h1>
<div class="meta">{runs} runs - {e(fw)} - {len(tests)} tests - generated {ts}</div>
<div class="cards">
  <div class="c score"><div class="k">FlakeScore</div><div class="val">{score}<span style="font-size:15px;color:var(--sub)"> / 100</span></div></div>
  <div class="c"><div class="k">Stable</div><div class="val" style="color:var(--ok)">{n["STABLE"]}</div></div>
  <div class="c"><div class="k">Flaky</div><div class="val" style="color:var(--warn)">{n["FLAKY"]}</div></div>
  <div class="c"><div class="k">Broken</div><div class="val" style="color:var(--bad)">{n["BROKEN"]}</div></div>
  <div class="c"><div class="k">To quarantine</div><div class="val">{n_q}</div></div>
</div>
<table>
  <thead><tr><th>Test</th><th>Results</th><th class="num">Pass</th><th class="num">Fail</th>
  <th class="num">Flakiness</th><th class="num">Flip</th><th>Verdict</th></tr></thead>
  <tbody>
{chr(10).join(trs)}
  </tbody>
</table>
<div class="formula">
  <code>flakiness rate = failures / attempts</code> &nbsp;-&nbsp;
  <code>flip rate = outcome changes / (attempts - 1)</code><br>
  Flagged FLAKY at &gt;= {detect*100:.0f}% - quarantine candidate at &gt;= {quarantine*100:.0f}%.
  A test that never passes is <b>BROKEN</b>, not flaky: that is a real failure, fix the test or the app.
  <code>FlakeScore = 100 x (1 - sum(flakiness of flaky tests) / total tests)</code>
</div>
<footer>Generated by the flaky-test-detector skill - The Testing Academy</footer>
</div></body></html>'''


def main():
    ap = argparse.ArgumentParser(description="Score flaky tests from collected run history.")
    ap.add_argument("--history", default=".flaky/history.jsonl")
    ap.add_argument("--html", metavar="FILE", help="also write an HTML report")
    ap.add_argument("--json", metavar="FILE", help="also write machine-readable JSON")
    ap.add_argument("--detect", type=float, default=DETECT_DEFAULT)
    ap.add_argument("--quarantine", type=float, default=QUARANTINE_DEFAULT)
    ap.add_argument("--no-color", action="store_true")
    ap.add_argument("--fail-on-flaky", action="store_true", help="exit 1 if any test is flaky (CI gate)")
    a = ap.parse_args()

    tests, meta = analyze(a.history, a.detect, a.quarantine)
    score = flake_score(tests)
    print(cli_table(tests, meta, score, a.detect, a.quarantine,
                    color=not a.no_color and sys.stdout.isatty()))

    if a.html:
        with open(a.html, "w") as fh:
            fh.write(html_report(tests, meta, score, a.detect, a.quarantine))
        print(f"HTML report: {a.html}")
    if a.json:
        payload = {
            "timestamp": datetime.datetime.now().isoformat(),
            "totalRuns": len([r for r in meta["runs"] if r is not None]),
            "totalTests": len(tests), "flakeScore": score,
            "flakyTests": [
                {"testName": t["id"], "file": t["file"], "flakinessRate": round(t["flakiness_rate"], 3),
                 "flipRate": round(t["flip_rate"], 3), "passes": t["passes"], "failures": t["failures"],
                 "quarantine": t["quarantine"], "failureMessages": [t["message"]] if t["message"] else []}
                for t in tests.values() if t["verdict"] == "FLAKY"],
            "brokenTests": [t["id"] for t in tests.values() if t["verdict"] == "BROKEN"],
        }
        with open(a.json, "w") as fh:
            json.dump(payload, fh, indent=2)
        print(f"JSON: {a.json}")

    if a.fail_on_flaky and any(t["verdict"] == "FLAKY" for t in tests.values()):
        sys.exit(1)


if __name__ == "__main__":
    main()

8. The CLI table

Two commands, real output. This is an actual run of the scripts above, 10 runs across a mixed Playwright + Selenium suite:

python scripts/collect.py --command "npx playwright test --reporter=json" --runs 5 --fresh
python scripts/analyze.py --html flaky-report.html

Flaky Test Report  10 runs - playwright, selenium/junit - 8 tests
-----------------------------------------------------------------------------------------------------
TEST                                                    RESULTS     PASS  FAIL   RATE   FLIP  VERDICT
-----------------------------------------------------------------------------------------------------
...ut.spec.ts > pay button enables after address entry  ✗✓✗✓✗✓✗✗✓✗     4     6    60%    89%  FLAKY +QUARANTINE
com.tta.tests.CheckoutTest.applyCouponUpdatesTotal      ✓✗✗✓✓✓✗✓✓✓     7     3    30%    44%  FLAKY
.../checkout.spec.ts > applies coupon code at checkout  ✓✗✓✓✓✗✓✓✓✓     8     2    20%    44%  FLAKY
com.tta.tests.GridTest.runsOnRemoteNode                 ✓✓✗✓✓✓✓✓✓✓     9     1    10%    22%  FLAKY
tests/cart.spec.ts > cart badge updates on add          ✓✓✓✓✓✓✓✓✓✗     9     1    10%    11%  FLAKY
tests/search.spec.ts > search returns results           ✗✗✗✗✗✗✗✗✗✗     0    10   100%     0%  BROKEN
com.tta.tests.LoginTest.loginWithValidUser              ✓✓✓✓✓✓✓✓✓✓    10     0     0%     0%  STABLE
...in.spec.ts > user can log in with valid credentials  ✓✓✓✓✓✓✓✓✓✓    10     0     0%     0%  STABLE
-----------------------------------------------------------------------------------------------------
FlakeScore: 83.8 / 100   2 stable  5 flaky  1 broken  1 to quarantine
flakiness rate = failures / attempts   detect >= 10%   quarantine >= 35%

HTML report: flaky-report.html

Read it left to right: the bead strip is the actual pass/fail history in order, so you can see the shape of the flake, not just the percentage. pay button enables after address entry flips almost every run (89%) which screams race condition, while cart badge updates on add failed once in ten and is probably noise until more runs say otherwise.

9. The HTML report

The same data as a self-contained file you can attach to a ticket, open in CI artifacts, or send to whoever asks "is the suite healthy?". No assets, no CDN, one file, and it respects dark mode.

It leads with the FlakeScore and the three counts, then one row per test with its bead strip, flakiness, flip rate and verdict, with the failure message inline so nobody has to go digging. The formula is printed at the bottom, because a number you cannot explain is a number that gets ignored.

10. Root cause, then quarantine

A percentage is not a fix. Once a test is flagged, the agent suggests a category from the failure message and the timing, and says unknown when the evidence is thin rather than inventing a story:

11. Wire it into CI

Run it nightly rather than on every PR: flakiness detection needs repetition, and you do not want to run your suite five times on every push. Weekly or nightly on main, with the report kept as an artifact, is the sweet spot.

name: nightly flaky scan
on:
  schedule: [{ cron: "0 2 * * *" }]   # 02:00 daily
  workflow_dispatch:
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx playwright install --with-deps
      - run: python scripts/collect.py --command "npx playwright test --reporter=json" --runs 5 --fresh
      - run: python scripts/analyze.py --html flaky-report.html --json flaky.json
      - uses: actions/upload-artifact@v4
        with:
          name: flaky-report
          path: flaky-report.html

Add --fail-on-flaky to analyze.py when you want the job to go red once flakiness appears. Start without it: on a suite that has never been measured, turning the gate on day one just teaches everyone to ignore a red job.

Get the skill

# Claude Code
cp -r flaky-test-detector ~/.claude/skills/

# then just ask
"run the suite 5 times and tell me which tests are flaky"

Want the rest of the set? This skill is part of the free 36-skill QA Skill Suite covering every phase of the STLC.