evaluate-skill

# Evaluate Skill Run a skill repeatedly to measure how reliably it works, and design the evals that measure it. ## Prerequisites The `caliper` CLI must be on `PATH`. This skill can be copied into an agent without the Caliper repo, so do not assume the CLI is packaged with it. Install if missing: ```bash pipx install caliper-eval ``` The engine (backend + model) is not part of the spec — it is chosen at run time with `--model` (skill) and `--judge-model` (judge), independently, from `claude-code`, `codex`, `pi`, defaulting to `claude-code`. Every backend is a CLI agent that uses its own subscription/auth; there is no direct-API backend (for API billing, configure a CLI with an API key). Full per-backend detail and every command: [REFERENCE.md](REFERENCE.md). ## Spec shape An `.eval.yaml` names the skill and a list of tasks. Keep `skill.path` relative to the spec file (usually `./SKILL.md`): ```yaml skill: path: ./SKILL.md # relative to the spec file tasks: - name: What success looks like prompt: <prompt sent to the skill under test> expect: <natural-language pass/fail criterion> assert: | # optional deterministic Python check assert ... ``` The spec has no `backend`/`model` or `judge:` block; pick the engine when you run, e.g. `caliper run <spec> --model codex --judge-model codex`. The full format (setup/cleanup, external assert scripts, sandbox) is in [REFERENCE.md](REFERENCE.md). ## Bundled references `references/evals/` holds ...