← ClaudeAtlas

testdoc-flakylisted

Finds flaky tests from the JUnit XML that CI already produces: collects several runs of the same suite, ranks tests that both passed and failed by flake rate, guesses the cause (timing, shared state, network, randomness, time zone or date, resource limits) from the failure text, and proposes the smallest fix for that cause, never a blanket retry. Use when the user says "this test is flaky", "tests fail randomly in CI", "find my flaky tests", "which tests fail intermittently", "it passes on rerun", or "rank flaky tests".
vivekbhange-hub/test-doctor-skill · ★ 0 · Testing & QA · score 75
Install: claude install-skill vivekbhange-hub/test-doctor-skill
# testdoc-flaky: which tests lie, and why A flaky test is a test that gives different answers for the same code. CI already holds the evidence: every run uploads a JUnit XML file. This skill collects several of them, lets `junit_report.py` do the counting, and then you do the reading: open the failing test, find the cause, and make the smallest change that removes it. ## Inputs - JUnit XML from at least 3 runs of the same suite, ideally 5-10, on the same commit or on commits that did not touch the failing code. One run cannot show flakiness. - Or access to the repo's CI through `gh` so you can download them. - Commands below say `python3`; on Windows use `python` (or `py -3`). ## Outputs - `testdoc/flaky-report.md`: the script's report (flaky, failing, slow, clusters). - A ranked list for the user: test, flake rate, likely cause with the evidence line, the fix. - Optionally one small change per flaky test, each in its own commit. ## Steps 1. **Confirm permission.** Only pull CI artifacts and run tests in repositories the user owns or is authorized to work on. Downloaded artifacts can contain secrets in logs; do not paste them. 2. **Get the runs.** If the user has no JUnit files yet, check which tool writes them (pytest `--junitxml=junit.xml`, jest-junit, Maven Surefire `target/surefire-reports`, Go `gotestsum --junitfile junit.xml`) and that the workflow uploads them with `actions/upload-artifact`. Then download recent runs of one workflow: ``` gh