pytest-optimizer-01-benchmarklisted
Install: claude install-skill tony/ai-workflow-plugins
# 01-benchmark
Prove or disprove each hypothesis by measurement. This phase applies candidates on
a **throwaway copy** (a scratch git worktree or a stash-guarded working tree),
never on committed history.
`$ARGUMENTS` may pass `--only=<heuristic-id>` to benchmark a subset, `--runs=N`
(measurement runs per candidate, default matches the baseline), and `--force` to
re-benchmark ids already recorded.
## Step 1: Load inputs
Read `baseline.json` (noise band, test command, capabilities) and
`hypotheses.json` from the resolved memory directory. If either is missing, tell
the user to run the `pytest-optimizer-00-scan` skill first. Substitute that resolved test
command wherever this file writes `pytest`. Skip any hypothesis whose
(content-derived) id already has a benchmark recorded **against the current
baseline token**, unless `--force`; a re-baseline after `03-execute` re-opens all
candidates for fresh measurement.
## Step 2: Per hypothesis — apply, measure, gate
For each open hypothesis, in isolation:
1. **Isolate.** Create a scratch worktree (or snapshot the working tree) so the
change can be applied and discarded without touching history.
2. **Apply** the single change the heuristic prescribes
(`references/heuristic-catalog.md`).
3. **Measure.** Run the suite serially, cache disabled, `--runs` times, recording
total wall-time:
```bash
pytest -p no:cacheprovider -p no:randomly -q
```
Compute the median delta (`baseline_median − candidate_median`) vs