← ClaudeAtlas

benchlisted

Formal benchmark harness. Runs a metric command N times across 2-3 variants (git refs or state-prep shell commands), checks variance, computes delta vs. declared baseline, and emits a reproducible TSV plus a one-paragraph summary. Read-only to source. Inspired by caveman's 3-arm eval.
arbazkhan971/godmode · ★ 26 · Code & Development · score 79
Install: claude install-skill arbazkhan971/godmode
## Activate When - `/godmode:bench`, "benchmark", "compare variants", "A/B metric" - User has a metric and wants a statistically-honest comparison between 2-3 code states - NOT for optimization loops — use `optimize` for that. `bench` measures, never modifies source. ## Inputs Ask once, cache for the session: - `metric_cmd` — shell command printing ONE number to stdout (lower-is-better or higher-is-better, user declares `direction`). - `variants` — list of 2-3 entries. Each variant is a record: - `name` — short label (e.g. `main`, `terse_on`, `feature_branch`) - `prep` — EITHER a git ref (`git checkout <ref>`) OR an inline shell command that puts the repo into the desired state (e.g. `export GODMODE_TERSE=1`, `git checkout feature-branch`) - `teardown` — optional shell command to undo `prep` (default: `git checkout -` for refs, `unset VAR` for env) - `baseline` — name of the variant all deltas are computed against. Must match one `variants[].name`. - `N` — runs per variant. Default 5. Minimum 3. Maximum 20. - `variance_threshold` — stdev/mean ratio that flags a variant noisy. Default 0.05 (5%). ## Workflow 1. Validate inputs — `metric_cmd` must emit a single number; `baseline` must match a variant; `N >= 3`. 2. Snapshot starting git state (`HEAD` sha, branch, dirty bit). Refuse to run if working tree is dirty. 3. FOR each variant in order: a. Run `prep`. Abort variant on non-zero exit, mark `prep_failed`. b. Run `metric_cmd` N times. Collect numbers into an arra