← ClaudeAtlas

flaky-detectlisted

Detect tests whose pass/fail verdict varies across repeated runs of unchanged code (flaky tests), from the JUnit XML / Cucumber JSON that run-report already produces. Flags with evidence only - never retries, quarantines, or fixes automatically. Use after collecting 3+ runs of the same suite.
QAIA-Project/QAIA · ★ 0 · Testing & QA · score 64
Install: claude install-skill QAIA-Project/QAIA
# flaky-detect — flaky test detection from repeated runs A test whose verdict flips between runs of unchanged code destroys the signal value of the whole suite: every red becomes negotiable, and a real regression hides behind "just re-run it". This skill names those tests with evidence, and stops there — it never retries, quarantines or fixes. The defect class is not hypothetical: QAIA's own automation has hit it (shared mutable state raced by parallel workers). For the provenance and the audited evidence trail, see `references/origin.md` — not needed to run the skill. Reference fixture: `fixture/` in this skill folder — a minimal, self-contained Playwright suite against a shared-state server, deliberately timing-dependent (not "always fails"), whose 5 captured real runs (`fixture/runs/results-run{1..5}.xml`) show genuine pass/fail variance per test with **zero code changes between runs**. See `fixture/VALIDATION.md` for the worked example of this skill applied to that data. ## Input - **N ≥ 3 runs** (recommended) of the **same test code**, same suite, in the formats `run-report` already produces: JUnit XML and/or Cucumber JSON. 2 runs is the visible floor — enough to see *a* difference, never enough to trust it isn't a one-off. **1 run can never show flakiness at all**: a single green run only proves "passed this once," never "is stable" — say so rather than implying otherwise. - The runs must span **no code change to the test or the SUT** between them — that is