strengthen-your-evalslisted
Install: claude install-skill whilehq/whileai-sdk
# Strengthen your evals
Every step below is in `check.py`, which runs offline in seconds. Its setup
defines the names the blocks use: `TOOLS`, `POLICY`, `OLD_TESTS` (the
three-ask suite the team has), `SEEDS` (one ask per policy branch),
`refund_judge` (the policy as a program), `VERSIONS` (`v1` the agent as
shipped, `v2` after a prompt fix; you have one entry today), `LABELS` (hand
labels for sixty replies), `MODEL`, and `fake` (a recording transport).
Your agent is any callable `message -> {"steps": [...], "final_text": ...}`
that runs its own tools; `wai init-evals` writes that wrapper.
## 1. Find what the suite never reaches
Ask what the old tests miss before writing new ones: which policy rule,
which tool, what stance, and whether any ask runs more than once.
```python
gap = wai.coverage_gap(OLD_TESTS, tools=TOOLS, system_prompt=POLICY)
print(wai.format_coverage_gap(gap))
```
A suite that passes on a good day only is the usual finding: every ask
ordinary, every ask run once, one rule no ask reaches.
## 2. Build the frozen test
The held-out set is one artifact, named, and the same for every version.
`seeds=` keeps your asks on the policy branches; the writer varies the
wording and the stance and adds asks of its own, and it reads the record
ids off your tool descriptions ("Orders on file: A1001, ..."). Write the
asks once, then replay that run with `tasks=` for every other version: the
writer steers toward what the agent it watches gets wrong, so two versions
run f