skill-eval-harnesslisted

Write, evaluate and maintain agent skills with measured evidence rather than assertion. Covers SKILL.md structure and frontmatter, writing a description that triggers reliably without firing on neighbouring topics, progressive disclosure through references and scripts, and building eval suites across three families - usefulness with skill on versus off, discovery, and adversarial discovery. Use when creating a new skill, reviewing or debugging one that does not trigger or does not help, deciding whether a skill earns its context cost, writing an eval suite or grader, or interpreting skill eval results. Also use when asked whether a skill is any good.
Premchand006/AI-assistant-skills · ★ 0 · Code & Development · score 64

Install: claude install-skill Premchand006/AI-assistant-skills

# Skills, measured Independent testing of published skill collections keeps finding the same thing: most skills do not help, and a meaningful share make output worse. The ones that help are written by someone who knows the domain and were iterated against evaluation. So the question for any skill is not whether it reads well, it is whether it beats the same model without it — and that is answerable. ## Order of work 1. Write the `description`. It is the only text loaded before selection, so it decides whether the skill ever runs. 2. Write the discovery evals from the description, before the body. This catches the description that reads well and selects nothing. 3. Write usefulness cases from real mistakes — something a model actually got wrong. A case where the skill-off answer is already correct measures nothing. 4. Write the scripts. Deterministic work belongs in a script, whose source never enters context. 5. Write `SKILL.md` aimed at the cases, and move depth to `references/`. 6. Run the evals. If the skill does not beat its own baseline, fix it or drop it. ```bash python3 scripts/new_skill.py my-skill-name # scaffold with evals in place python3 ../../tools/validate_skills.py # frontmatter, budgets, style python3 ../../evals/eval_runner.py --skill my-skill-name --runner cli --record python3 ../../evals/eval_runner.py --runner replay # re-grade, no API key ``` ## The three eval families **Usefulness** runs the same prompt with the sk