← ClaudeAtlas

flaky-testlisted

Hunt down a test that passes sometimes and fails sometimes: measure its failure rate by rerunning, then isolate the cause (test order, shared state, time, randomness, concurrency, network, the machine) one factor at a time, fix the cause and prove the rate is zero. Use for "flaky test", "intermittent failure", "fails in CI but passes locally", "passes on retry", or before adding a retry or skip to a test.
ychampion/ystack · ★ 4 · AI & Automation · score 77
Install: claude install-skill ychampion/ystack
# Flaky test A retry hides a flaky test; it doesn't fix it, and sometimes the flake is a real bug users hit. Turn "sometimes" into a number, then move that number to zero. ## When it applies - A test fails without any related code change, then passes on rerun. - It fails in CI and never locally, or the other way round. - Someone proposes a retry, a longer timeout or a skip. ## Procedure ### 1. Measure the rate Run the single test in a loop, at least 50 times, and count failures. If the runner can't pick one test, loop the whole suite. Save each failure's output in a scratch folder, not the repo. ```bash for i in $(seq 1 50); do <test command> > "$TMP/run-$i.log" 2>&1 || echo "fail $i"; done ``` Zero failures alone? Run it inside its whole file, then the whole suite, then with CI's parallelism. Now you know which context makes it flake. Compare the failure messages: one message or several? Note which test fails; the cause is often a different test that ran before it. ### 2. Isolate one factor at a time Check each, changing only that factor, and rerun the loop: | Factor | How to test it | | --- | --- | | Test order / shared state | Run with a random or reversed order, or the test alone vs after its neighbour. Leftover globals, singletons, DB rows, temp files, env vars, mocks not restored. | | Time | Freeze or shift the clock; run near midnight, month end, DST, another timezone. | | Randomness | Seed the generator; try several seeds and find one that fails every time.