eval-skills

Solid

Eval and improve a skill against golden cases — run the target skill blind in a fresh, context-free subagent on each example input, grade the artifact against the expected outcome, and let the gaps drive the edits. Use when the user wants to test/eval/improve/harden a skill, says "this skill keeps producing X / keeps missing Y", or hands a skill plus example input→expected-output pairs. Pairs with [write-skills](../write-skills/SKILL.md) (the authoring principles every fix obeys).

AI & Automation 41 stars 4 forks Updated 2 days ago Apache-2.0

Install

View on GitHub

Quality Score: 84/100

Stars 20%
54
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Eval Skills Treat a skill like a function under test. Feed it example inputs in a clean room, check the artifacts against what good looks like, and let the failures drive the edits. The eval is only honest if the run is **blind**: the agent executing the skill must carry none of this conversation's context and must never see the expected output. Leak either and you are teaching to the test. ## Inputs you need — refuse without them Confirm all three before spawning anything. If any is missing or unresolvable, stop and tell the user *exactly which one* and what a good version looks like. Do not invent cases, guess intent, or eval against a fuzzy wish. - **Target skill** — must resolve to a real `SKILL.md`. If you can't find it, list the skills you can see and ask which one they mean. - **At least one golden case** — a concrete input the skill will actually receive: a screenshot, a prompt, a file, a scene. "Improve write-spec" with no input attached is not a case. - **The bar per case** — the outcome a good artifact achieves and the smells that would make it bad, *not* an exhaustive parts list. The skill's **judgment** is what's under test, so do not pre-enumerate every requirement — that turns the eval into a conformance check and stops testing whether the skill decides well. "Sliced so each piece is independently buildable and verifiable, at the granularity a competent practitioner would pick — a lazy mega-slice and pointless over-splitting are both fai...

Details

Author
dzhng
Repository
dzhng/duet-agent
Created
3 months ago
Last Updated
2 days ago
Language
TypeScript
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category