← ClaudeAtlas

watchlisted

Keep the platform's Live tile honest with one command a day, without the platform ever ingesting a trace. Use when an agent is served and someone asks "is it still fine in production", when the Live tile is empty or stale, when a served version needs a flagged rate that means the same thing as its held-out score, or when deciding whether a rise in failures is real enough to open a new run. Covers reading yesterday's traces, a date-seeded sample scored by the held-out judge, posting one LiveDay, the two-day drift rule against the behavior's noise floor, and scheduling.
whilehq/whileai-sdk · ★ 3 · AI & Automation · score 77
Install: claude install-skill whilehq/whileai-sdk
# Watch the served version The held-out score says what a version can do. The Live tile says whether that survived real traffic (`LiveDay`: the flagged rate over time is the only number that does). Traces never leave your side: you read them, you score a sample with the same judge the held-out test uses, and you post one row per day. Names in the blocks below come from `check.py`: `served_bot` is the scripted agent as served (v3), `refund_judge` is the held-out judge (a program), `TOOLS`, `ASKS`, `URGENT_ASKS`, `OIDS` define its world, `trace_path(day)` says where a day's export lives, and `fake` is the offline transport. ## 0. Declare the behavior and score the served version once The daily job compares against two numbers the platform already holds: the behavior's `noise_floor` and the served version's held-out score. Declare the first; post the second once per served version. The judge here is the one the held-out test uses; the sample is scored by the same function. ```python tracked = track("refund-bot", model="Qwen/Qwen3-4B", transport=fake) # drop transport= for real tracked.behavior( Behavior( name="refunds", test_version="v1", judge=Judge(name="refund policy as a program"), noise_floor=2.0, # points; the re-run spread you measured on this test reward_is_judge=False, description="Looks the order up, refunds only when policy allows, escalates over-limit.", ) ) HELDOUT = [a.format(oid=o) for a in ASKS for o