flaky-test-triagelisted

# Flaky test triage A flaky test is a **defect report you haven't read yet**. Roughly half the time the defect is in the test; the other half it is a real race, a real leak, or a real ordering assumption in the system — the exact bugs that are hardest to find any other way. **Never make retries the fix.** Auto-retry converts a signal into silence: the race is still in production, and now the suite lies. Retries are acceptable only as a temporary measure attached to a ticket with an owner and a date. ## The protocol ### 1. Capture before it disappears The moment you see a flake, save the evidence — the next run overwrites it: full failure output, the seed/order if randomized, the CI job, the commit, timestamp, and whether the machine was loaded. Flakes that "went away" almost always come back at the worst moment. ### 2. Quarantine, don't ignore If it blocks the pipeline, move it out of the blocking suite — but with a rule: an owner and a deadline, recorded where work is tracked. A quarantine directory with no owner becomes a graveyard, and the bug it was reporting ships. If nobody will own it, **delete the test**. An unowned quarantined test provides zero signal and non-zero maintenance. Deleting is at least honest. ### 3. Reproduce deliberately Do not wait for it to happen again. Force it: ```bash # many times in a row, stop on first failure for i in $(seq 200); do <run one test> || break; done # randomize order — catches shared state and order dependence <runner>