← ClaudeAtlas

flake-quarantinelisted

Auto-tag tests that flaked >N times in M runs, move them to a quarantine suite, and cap how long they stay there
bakw00ds/yakos · ★ 2 · AI & Automation · score 81
Install: claude install-skill bakw00ds/yakos
# Flake Quarantine ## Purpose Keep the main test suite green by sweeping flaky tests into a quarantine suite. Used by `test-runner`. The skill reads recent CI test results, identifies tests that failed intermittently above a threshold, tags them as quarantined, and reports on quarantined tests that have lingered past their cap (signal: someone needs to fix or delete them). ## Scope - Reads CI test-result history (JUnit XML, JSON, or whatever the project's runner emits) for the last M runs across `main` and short-lived branches. - Identifies tests with flake-rate above `--threshold` (default `3/20` = flaked 3+ times in 20 runs). - Adds a `@quarantined` tag (or framework equivalent — see Manual pass) to each flagged test, with a comment recording the date quarantined and the observed flake-rate. - Reports quarantined tests that have lingered past `--max-quarantine-days` (default 14) — these are stale and need human attention. - The CI pipeline runs the quarantine suite as a non-blocking job — the flakes still execute, but they don't fail the build. ## When to use - As a weekly maintenance pass on the test suite. - After a CI outage caused by a single flaky test taking down unrelated PRs — sweep, then strengthen the threshold. - Before a release branch cut — confirm no high-flake-rate tests are in the main suite. ## When NOT to use - As a substitute for fixing flakes. Quarantine is a triage hospital, not a graveyard. The `--max-quarantine-days` cap