flaky-testlisted
Install: claude install-skill ychampion/ystack
# Flaky test
A retry hides a flaky test; it doesn't fix it, and sometimes the flake is a real bug users hit. Turn
"sometimes" into a number, then move that number to zero.
## When it applies
- A test fails without any related code change, then passes on rerun.
- It fails in CI and never locally, or the other way round.
- Someone proposes a retry, a longer timeout or a skip.
## Procedure
### 1. Measure the rate
Run the single test in a loop, at least 50 times, and count failures. If the runner can't pick one
test, loop the whole suite. Save each failure's output in a scratch folder, not the repo.
```bash
for i in $(seq 1 50); do <test command> > "$TMP/run-$i.log" 2>&1 || echo "fail $i"; done
```
Zero failures alone? Run it inside its whole file, then the whole suite, then with CI's parallelism.
Now you know which context makes it flake. Compare the failure messages: one message or several?
Note which test fails; the cause is often a different test that ran before it.
### 2. Isolate one factor at a time
Check each, changing only that factor, and rerun the loop:
| Factor | How to test it |
| --- | --- |
| Test order / shared state | Run with a random or reversed order, or the test alone vs after its neighbour. Leftover globals, singletons, DB rows, temp files, env vars, mocks not restored. |
| Time | Freeze or shift the clock; run near midnight, month end, DST, another timezone. |
| Randomness | Seed the generator; try several seeds and find one that fails every time.