tool-call-efficiencylisted
Install: claude install-skill whilehq/whileai-sdk
# Tool-call efficiency
Every step below is in `check.py`, which runs offline in seconds. Its setup
defines the names the blocks use: `TOOLS`, `POLICY`, `before_agent` (wastes
calls), `after_agent` (does not), `HOLDOUT_SEEDS`, `TRAIN_SEEDS`, `K`,
`N_TASKS`, `REWARD_REF`, `stamp` (puts each task's checks in
`privileged.reference`), `pct`, `half`, `fake` (a recording transport), and
`tool_calls` from `whileai.simulations.score.hygiene`.
## 1. The score is a program
**Solved within budget.** A task counts when its checks pass AND the rollout
spent at most `BUDGET` tool calls. The behavior's score is the share of
held-out tasks solved within budget; higher is better. Mean calls per solved
task is the second number, from the same rows. No model grades anything, so
this is a verifiable reward (the SDK's `whileai.simulations.verify` module,
and "Policy Gradients", where GRPO climbs a reward that is a checker), and
`reward_is_judge=False` holds by construction.
```python
BUDGET = 1 # tool calls a solved task may spend
@wai.verifier
def solved(candidate, reference, row):
"""The task's checks: the reply says what the record says. Budget ignored."""
return reference.lower() in candidate.lower()
@wai.verifier
def within_budget(candidate, reference, row):
"""Solved AND at most BUDGET tool calls. No judge: a program decides."""
if reference.lower() not in candidate.lower():
return 0, "unsolved"
calls = tool_calls(row)
if calls > BUDGET:
retur