skill-eval-looplisted

Prove whether a candidate skill actually improves agent performance before adopting it - blind A/B evals on reconstructed real tasks, a three-tier adoption gate, and production rechecks. Use when asked to "eval this skill", "does this skill help", "test this skill before installing", "run the auto-improve loop", or "recheck probationary skills". Takes candidates from skill-miner; runs locally with the user's own agent CLI and models.
getedgehq/skills · ★ 3 · AI & Automation · score 71

Install: claude install-skill getedgehq/skills

# skill-eval-loop The proof half of the auto-improve loop. skill-miner finds pain and candidates; this skill decides what earns a place in the fleet. Everything runs locally - the user's own machine, own agent CLI, own models, own tasks. ## The loop ```bash python3 scripts/brief.py <clusters.json> --index 0 [--knowledge-gap] # theme -> eval brief for n in 1 2 3; do FORGE_SAMPLE=$n bash scripts/run_eval.sh <brief.json> without "" # baseline arm # ...or <incumbent> instead of "" to make the baseline the skill already # installed for this trigger, which is what adoption competes with FORGE_SAMPLE=$n bash scripts/run_eval.sh <brief.json> with <skill> # candidate arm python3 scripts/judge.py <brief.json> --sample $n # blind A/B + verify done python3 scripts/aggregate.py <brief.json> # strict majority python3 scripts/gate.py <brief.json> <skill> [--adopt-dir DIR] [--probation --failure-rate R] python3 scripts/recheck.py --sources claude,opencode,codex --dry-run # weekly review, decides nothing ``` Or hands-free end to end: ```bash bash scripts/forge.sh --mode corrections --memory ~/.claude/projects/<proj>/memory # recommended bash scripts/forge.sh --mode corrections --dry-run # stop before eval spend bash scripts/forge.sh --brief briefs/x.json --skill-dir drafts/y --samples 3 # eval only bash scripts/forge.sh --brief briefs/x.json --skill-dir drafts/y --rival ~/.claude/skills/z ``` `forge.sh`: