evallisted

# /eval — Skill Evaluation Skills need tests like code does. A modified skill that silently degrades is worse than no skill — it creates false confidence. This protocol defines how to write, run, and report skill evals. ## When to Run - After modifying a skill's protocol or constraints - During periodic review (scout Phase 6) - When a model upgrade may make a skill redundant - On request, to validate skill effectiveness ## EVAL.md Format Each skill can have a companion `EVAL.md` in its directory. Structure: ```markdown --- skill: <skill-name> --- # Eval: /pds:<skill-name> ## Scenarios ### Scenario: [descriptive name] **Setup:** [Context — files present, task state, branch state] **Prompt:** [The situation that should trigger this skill] **Expected:** - [ ] [Observable behavior — binary, not subjective] - [ ] [Another expected behavior] **Anti-patterns:** - [ ] [Thing the agent should NOT do] ## Baseline [Behavior WITHOUT the skill — for A/B comparison] ``` **Rules for scenarios:** - Expected behaviors are binary: did it or didn't. No "good enough." - Anti-patterns catch the failure modes the skill was created to prevent. - 2-3 scenarios per skill. Cover the happy path and the key failure mode. ## Execution Protocol ### Manual Eval 1. Read the skill's `EVAL.md` 2. For each scenario: spawn a worker with the setup and prompt 3. Observe worker behavior against the expected checklist 4. Record results ### Scout Eval (Phase 6) Scout reads EVAL.md for skills exercised d