← ClaudeAtlas

tool-call-efficiencylisted

Measure and train down the number of tool calls an agent spends per solved task, with no judge. Use when a coding agent is asked to make an agent cheaper or faster without making it worse: redundant re-fetches, tools called that the task did not need, lookups repeated across turns. The score is a program (checks pass AND at most `budget` calls), so it is a verifiable reward for GRPO, and correctness is tracked beside it so cutting calls never cuts solves.
whilehq/whileai-sdk · ★ 3 · AI & Automation · score 77
Install: claude install-skill whilehq/whileai-sdk
# Tool-call efficiency Every step below is in `check.py`, which runs offline in seconds. Its setup defines the names the blocks use: `TOOLS`, `POLICY`, `before_agent` (wastes calls), `after_agent` (does not), `HOLDOUT_SEEDS`, `TRAIN_SEEDS`, `K`, `N_TASKS`, `REWARD_REF`, `stamp` (puts each task's checks in `privileged.reference`), `pct`, `half`, `fake` (a recording transport), and `tool_calls` from `whileai.simulations.score.hygiene`. ## 1. The score is a program **Solved within budget.** A task counts when its checks pass AND the rollout spent at most `BUDGET` tool calls. The behavior's score is the share of held-out tasks solved within budget; higher is better. Mean calls per solved task is the second number, from the same rows. No model grades anything, so this is a verifiable reward (the SDK's `whileai.simulations.verify` module, and "Policy Gradients", where GRPO climbs a reward that is a checker), and `reward_is_judge=False` holds by construction. ```python BUDGET = 1 # tool calls a solved task may spend @wai.verifier def solved(candidate, reference, row): """The task's checks: the reply says what the record says. Budget ignored.""" return reference.lower() in candidate.lower() @wai.verifier def within_budget(candidate, reference, row): """Solved AND at most BUDGET tool calls. No judge: a program decides.""" if reference.lower() not in candidate.lower(): return 0, "unsolved" calls = tool_calls(row) if calls > BUDGET: retur