← ClaudeAtlas

ab-testing-whether-skills-still-triggerlisted

A/B whether skills still fire after changing skillOverrides or descriptions, using real prompts mined from session logs: recall, wrong skill, false fires, start tokens, Wilson CI + McNemar. Triggers on: do my skills still trigger, A/B skill overrides, test name-only skills, skill fires on the wrong prompt, hidden skill makes prompts fire nothing, rep-level paired bound.
sblattj/agent-chandlery · ★ 0 · Testing & QA · score 72
Install: claude install-skill sblattj/agent-chandlery
## More triggers skill-trigger-ab (old name), synth only for these skills, mined positives too thin, rename arm keeps each skill's state. # ab-testing-whether-skills-still-trigger Proves a skill-visibility change (name-only, hidden, a reworded description) didn't break which skill Claude picks. ## Do this Scripts live in `scripts/` (shared helper `lib/ccpaths.py`). Full flag docs: `scripts/README.md`, or `-h` on any script. Self-tests (no network): `cd scripts && python3 -m unittest discover -s tests`. ```bash P=${SKILL_DIR:-~/.claude/skills/ab-testing-whether-skills-still-trigger}/scripts; W=${AB_WORK:-~/skill-ab}/ab-$(date +%m%d); mkdir -p $W python3 $P/mine.py ~/.claude/projects --all-projects --out $W/candidates.jsonl --days 30 --max-negatives 150 # interactive sessions only python3 $P/filter.py $W/candidates.jsonl --out $W/cases.jsonl --model claude-haiku-5-5 # Haiku 5.5 judge (--model required, bare haiku rejected) keeps standalone prompts python3 $P/make_arms.py --skills-dir ~/.claude/skills --out $W/arms --base-settings ~/.claude/settings.json \ --arm current --arm after --set 'after:some-skill=name-only' # first --arm = baseline; ARM:GLOB=STATE, in order python3 $P/run.py --cases $W/cases.jsonl --arms $W/arms --out $W/results.jsonl --reps 2 --workers 6 --model sonnet # resumable; tail -f $W/results.jsonl.progress python3 $P/report.py $W/results.jsonl # Wilson 95% CI + paired McNemar vs the first arm ``` Optional: `sy