flaky-test-jurylisted

# Flaky Test Jury ## 1. System Architecture & Prerequisites Flaky tests randomly pass/fail and quietly erode CI reliability. The Jury ingests N reruns of the same test suite (JUnit XML or plain text logs), correlates each test against its pass/fail history, classifies flakes into a taxonomy, and emits a report that tells you *what type* of flake it is and *how to fix that class specifically*. Python 3.9+ stdlib. Flake taxonomy: - `EXT_TIMING` — sleeps/waits raced by CI load - `EXT_ORDER` — shared mutable state across tests (isolation leak) - `EXT_RESOURCE` — port/DB/filesystem collisions - `EXT_ASYNC` — unawaited promises / missing await - `EXT_FLAKY_NETWORK` — external calls without retry - `EXT_DETERMINISTIC` — always fails; not a flake, a bug ## 2. Input/Output Data Contracts Input: a directory containing JUnit-style `*.xml` reports from repeated runs (each file = one run). Optional `.testsuite` metadata (duration). Output: `flaky-jury-report.md` + `flaky-jury.json`: ```json { "runs": 5, "tests": 412, "flakes": 9, "perClass": { "EXT_TIMING": 4, "EXT_ORDER": 3, "EXT_ASYNC": 2 }, "hotspots": [ { "test": "auth.spec.ts::login-retry", "runs": 5, "passes": 3, "class": "EXT_TIMING", "evidence": ["2026-09-10 12:00:01.123 WARN retry", "timeout after 5000ms"] } ] } ``` Exit code: `0` = no flakes, `1` = flakes found. ## 3. Production Reference Implementation ```python #!/usr/bin/env python3 """flaky_test_jury.py — classify CI flaky tests from repeated run logs.""" import