← ClaudeAtlas

agent-benchmarklisted

Measure which executor does a kind of task best on this machine (Claude Code, OpenCode with a free model, a local Ollama model, or a program without AI) by time to correct and verified code, with repeated runs, percentiles, hardware and versions, and use the history to route future work. Also checks claims about model speed, memory, VRAM or energy. Use it when the owner asks which model or agent to use, why OpenCode or Ollama feels slow, whether delegating is worth it, or wants to compare models, even if they only ask "which one is faster".
SVN11X/skills · ★ 0 · AI & Automation · score 72
Install: claude install-skill SVN11X/skills
# agent-benchmark The best executor is the one that reaches verified, correct code soonest, including retries. First-token latency and model size are not the measure. ## Look before measuring `python <standard>/scripts/orchestrate.py doctor --full` shows OpenCode, its models, Ollama, the models loaded and their share on the GPU, the VRAM in use and the shells. A model split between GPU and CPU is slow on its own, so check `ollama ps` before blaming anything else. Never pull a model or change a timeout without the owner. ## Measure - Fixing code: `python <standard>/scripts/orchestrate.py bench --task fix-bug --executors opencode:<model> opencode:<other> --runs 3`. Each run starts from an identical synthetic repository, the executor works in its own worktree, and hidden acceptance tests it never saw decide whether the result counts. - Short local work: `python <standard>/scripts/orchestrate.py bench --task classify --executors tools ollama:qwen3.5:4b --runs 3`. The keyword baseline without AI is always in the comparison. - Other classes: `--task` also takes `extract`, `summarize`, `web-subprofile`, `tool-call`, `vision` and `design-judge` on Ollama, and `fix-multi` and `docs-small` on a coding agent. A program grades every one. - Many models at once: `python <standard>/scripts/analyze.py prepare`, then `plan`, then `run`. It tests every candidate of `settings/models.yml` that is installed and allowed, one local model at a time, and can be stopped and continued. `analyze.py