instruction-eval

# Condition A/B Change one condition an agent runs under, run the same prompts before and after, and show the difference. The conditions surrounding an agent have no verification. You can read code and tests will catch a regression, but a few lines added to instructions or a reference doc dropped in a directory only ever get judged on whether they sound reasonable. Even the person who put them there has no idea whether they change behavior. This skill replaces that guess with an observation. This runs on Claude Code. Both arms execute as `claude -p` subprocesses, so the CLI has to be available. ## Who writes what The report holds content from two sources, visually separated in the HTML. Never hand-write what the script produces, since transcribing only introduces errors. | Automatic (scripts) | LLM (`insights.json`) | |---|---| | Experiment setup: n, model, both directory paths | What you changed | | Prompt text | The claim about what changes | | Answer text (every arm, every rep) | Differences observed per prompt | | Metric medians and deltas | Metric interpretation | | Run anomalies (errors, permission denials) | Conclusion | ## Procedure ### 1. Design and build the conditions **Pick the axis first.** What you compare against what determines everything else. Only what the user mentioned is a candidate, and even when it reads as obvious you confirm it in step 2. Anything can be the axis: instruction text, reference material instructions point at, MCP tools and perm...