← ClaudeAtlas

skill-evaluatorlisted

Rigorously evaluate an Agent Skill end-to-end across ANY coding-agent CLI — verify its scripts emit the documented numbers (deterministic checks), test whether its description triggers on the right prompts, and measure whether an agent following the SKILL.md beats a no-skill baseline (with/without pass-rate delta, mean ± stddev, benchmarked). Use whenever you need to test, benchmark, validate, grade, or quantify a skill's quality, check if a skill "actually works," compare two skill versions, optimize a skill's triggering, or set up an eval suite — even if the user just says "is this skill any good," "does my skill work," or "benchmark this skill." Drives Claude Code, OpenAI Codex, Antigravity (agy), Cursor, GitHub Copilot, Amp, opencode, or Grok in headless mode.
HeshamFS/materials-simulation-skills · ★ 93 · AI & Automation · score 88
Install: claude install-skill HeshamFS/materials-simulation-skills
# Skill Evaluator Test whether a skill is **correct, discoverable, and valuable** — not just whether its unit tests pass. The harness is agent-agnostic: it drives whichever coding-agent CLI the user uses, because Agent Skills are portable across all of them. ## When to use which layer Three layers, increasing cost and fidelity (full rationale in `references/methodology.md`): | Layer | Question | Script | Needs a CLI? | |------|----------|--------|--------------| | 1. Deterministic | Do the scripts emit the **documented** numbers? | `run_script_checks.py` | No | | 2. Trigger | Does the **description** activate on the right prompts? | `run_trigger_eval.py` | Yes | | 3. Quality | Does following the SKILL.md **beat no skill**? | `run_quality_eval.py` → grade → `aggregate_benchmark.py` | Yes | Always run Layer 1 (it's free). Add Layers 2–3 when you can run a coding-agent CLI. ## Step 0 — pick the agent CLI Ask the user which coding agent they use, then map it to an adapter id. Supported: `claude-code`, `openai-codex`, `antigravity` (the `agy` CLI that replaced Gemini CLI on 2026-06-18), `cursor-cli`, `github-copilot-cli`, `amp`, `opencode`, `grok-cli`. See the full matrix and auth in `references/adapters.md`, or run: ```bash python scripts/agent_adapters.py list ``` Confirm the binary is installed and the auth env var is set (the matrix lists it). **Before any real run, dry-run it** to see the exact command: ```bash python scripts/agent_adapters.py build <agent> --prompt