skill-actually-workslisted

# Does the skill do anything People install forty skills and can tell you what none of them changed. A skill that never fires, or fires and produces what the model would have produced anyway, is occupying context and attention for nothing. This is the same question as "does the model matter", asked one layer up, and it gets answered the same way: by ablation. ## The refusal **Do not credit a skill with an improvement measured on one example.** If the on/off comparison has too few tasks to distinguish from noise, the answer is "no measurable effect on n tasks", not "it works". ## Procedure 1. **Collect 15 to 30 real tasks** the skill claims to cover. Real ones, from actual work. Tasks written to suit the skill will flatter it. 2. **Run each task twice**, with the skill available and with it removed. Same model, same phrasing, same everything else. Order matters less than keeping it constant. 3. **Check it fired at all.** A large share of skills never trigger, because the description does not match how anyone phrases the request. If it did not fire, that is the finding, and the fix is the description, not the body. 4. **Grade the pairs blind.** Whoever grades should not know which output had the skill. Self-grading a skill you wrote is not evidence. 5. **Use a paired test.** The same tasks went through both conditions, so compare the disagreements, not the totals. See `model-bakeoff` for the machinery. 6. **Separate three outcomes.** Changed and bet