agent-benchmarklisted
Install: claude install-skill SVN11X/skills
# agent-benchmark
The best executor is the one that reaches verified, correct code soonest, including retries. First-token latency and model size are not the measure.
## Look before measuring
`python <standard>/scripts/orchestrate.py doctor --full` shows OpenCode, its models, Ollama, the models loaded and their share on the GPU, the VRAM in use and the shells. A model split between GPU and CPU is slow on its own, so check `ollama ps` before blaming anything else. Never pull a model or change a timeout without the owner.
## Measure
- Fixing code: `python <standard>/scripts/orchestrate.py bench --task fix-bug --executors opencode:<model> opencode:<other> --runs 3`. Each run starts from an identical synthetic repository, the executor works in its own worktree, and hidden acceptance tests it never saw decide whether the result counts.
- Short local work: `python <standard>/scripts/orchestrate.py bench --task classify --executors tools ollama:qwen3.5:4b --runs 3`. The keyword baseline without AI is always in the comparison.
- Other classes: `--task` also takes `extract`, `summarize`, `web-subprofile`, `tool-call`, `vision` and `design-judge` on Ollama, and `fix-multi` and `docs-small` on a coding agent. A program grades every one.
- Many models at once: `python <standard>/scripts/analyze.py prepare`, then `plan`, then `run`. It tests every candidate of `settings/models.yml` that is installed and allowed, one local model at a time, and can be stopped and continued. `analyze.py