← ClaudeAtlas

pytest-optimizer-01-benchmarklisted

Phase 2 of the pytest-optimizer pipeline. Test each hypothesis from 00-scan in isolation on a throwaway working copy and re-measure its wall-clock delta against the project's noise band. Runs the safety gates (order independence, collection determinism, green re-run) per candidate, rejects anything within noise or failing a gate, and records the measured delta, confidence, and observed risk to benchmarks.json. Never mutates committed history. Idempotent per hypothesis id. Use after 00-scan to find out which proposed speedups are real.
tony/ai-workflow-plugins · ★ 2 · Testing & QA · score 59
Install: claude install-skill tony/ai-workflow-plugins
# 01-benchmark Prove or disprove each hypothesis by measurement. This phase applies candidates on a **throwaway copy** (a scratch git worktree or a stash-guarded working tree), never on committed history. `$ARGUMENTS` may pass `--only=<heuristic-id>` to benchmark a subset, `--runs=N` (measurement runs per candidate, default matches the baseline), and `--force` to re-benchmark ids already recorded. ## Step 1: Load inputs Read `baseline.json` (noise band, test command, capabilities) and `hypotheses.json` from the resolved memory directory. If either is missing, tell the user to run the `pytest-optimizer-00-scan` skill first. Substitute that resolved test command wherever this file writes `pytest`. Skip any hypothesis whose (content-derived) id already has a benchmark recorded **against the current baseline token**, unless `--force`; a re-baseline after `03-execute` re-opens all candidates for fresh measurement. ## Step 2: Per hypothesis — apply, measure, gate For each open hypothesis, in isolation: 1. **Isolate.** Create a scratch worktree (or snapshot the working tree) so the change can be applied and discarded without touching history. 2. **Apply** the single change the heuristic prescribes (`references/heuristic-catalog.md`). 3. **Measure.** Run the suite serially, cache disabled, `--runs` times, recording total wall-time: ```bash pytest -p no:cacheprovider -p no:randomly -q ``` Compute the median delta (`baseline_median − candidate_median`) vs