watchlisted
Install: claude install-skill whilehq/whileai-sdk
# Watch the served version
The held-out score says what a version can do. The Live tile says whether that
survived real traffic (`LiveDay`: the flagged rate over time is the only number
that does). Traces never leave your side: you read them, you score a sample
with the same judge the held-out test uses, and you post one row per day.
Names in the blocks below come from `check.py`: `served_bot` is the scripted
agent as served (v3), `refund_judge` is the held-out judge (a program),
`TOOLS`, `ASKS`, `URGENT_ASKS`, `OIDS` define its world, `trace_path(day)`
says where a day's export lives, and `fake` is the offline transport.
## 0. Declare the behavior and score the served version once
The daily job compares against two numbers the platform already holds: the
behavior's `noise_floor` and the served version's held-out score. Declare
the first; post the second once per served version. The judge here is the
one the held-out test uses; the sample is scored by the same function.
```python
tracked = track("refund-bot", model="Qwen/Qwen3-4B", transport=fake) # drop transport= for real
tracked.behavior(
Behavior(
name="refunds",
test_version="v1",
judge=Judge(name="refund policy as a program"),
noise_floor=2.0, # points; the re-run spread you measured on this test
reward_is_judge=False,
description="Looks the order up, refunds only when policy allows, escalates over-limit.",
)
)
HELDOUT = [a.format(oid=o) for a in ASKS for o