← ClaudeAtlas

strengthen-your-evalslisted

Turn the tests an agent already has into an eval that can fail, measure it the way a paper would, and report it so the platform can tell a fix from a fluctuation. Use when a coding agent is asked to build or improve evals for an agent that runs on a frontier model (Claude, GPT, Gemini) or on its own weights: find what the suite never reaches, build a frozen held-out set, check the judge against people, read pass@1 with an interval and the failure-capable count, measure the noise floor, size the set, and post every behavior to while.ai/platform/runs. No GPU, no key until you want the hosted writer.
whilehq/whileai-sdk · ★ 3 · AI & Automation · score 77
Install: claude install-skill whilehq/whileai-sdk
# Strengthen your evals Every step below is in `check.py`, which runs offline in seconds. Its setup defines the names the blocks use: `TOOLS`, `POLICY`, `OLD_TESTS` (the three-ask suite the team has), `SEEDS` (one ask per policy branch), `refund_judge` (the policy as a program), `VERSIONS` (`v1` the agent as shipped, `v2` after a prompt fix; you have one entry today), `LABELS` (hand labels for sixty replies), `MODEL`, and `fake` (a recording transport). Your agent is any callable `message -> {"steps": [...], "final_text": ...}` that runs its own tools; `wai init-evals` writes that wrapper. ## 1. Find what the suite never reaches Ask what the old tests miss before writing new ones: which policy rule, which tool, what stance, and whether any ask runs more than once. ```python gap = wai.coverage_gap(OLD_TESTS, tools=TOOLS, system_prompt=POLICY) print(wai.format_coverage_gap(gap)) ``` A suite that passes on a good day only is the usual finding: every ask ordinary, every ask run once, one rule no ask reaches. ## 2. Build the frozen test The held-out set is one artifact, named, and the same for every version. `seeds=` keeps your asks on the policy branches; the writer varies the wording and the stance and adds asks of its own, and it reads the record ids off your tool descriptions ("Orders on file: A1001, ..."). Write the asks once, then replay that run with `tasks=` for every other version: the writer steers toward what the agent it watches gets wrong, so two versions run f