← ClaudeAtlas

dpo-pairslisted

Turn graded rollouts of an agent into chosen/rejected preference pairs for DPO, check that the pairs teach the behavior and not a tic, write the JSONL a DPO trainer loads, and prove the result on a frozen held-out test reported to the While platform. Use when a coding agent is asked to build DPO or preference data from an agent's own rollouts, to fix a tool-use or policy slip with preference training, or to show that a trained version beats the served one.
whilehq/whileai-sdk · ★ 3 · AI & Automation · score 77
Install: claude install-skill whilehq/whileai-sdk
# DPO pairs from graded rollouts DPO trains on pairs: same ask, one reply preferred, no reward model in the loop ("Direct Alignment"). Where the same ask passed once and failed once, the contrast is the signal. Names from `check.py` used below: `TOOLS` and `POLICY` (a refund agent's tools and rules), `TRAIN_ASKS` and `TEST_ASKS` (two disjoint ask lists), `before` (the agent you have; a scripted bot that refunds before looking the order up on half its rollouts), `after` (stands in for the model trained on the pairs), `refund_judge` (the policy as a program), `train_reward` (one rule of it, the training signal), `length_judge` (an untrained behavior), `fake` (a recording transport), `OUT` (an output folder). ## 1. Freeze the held-out test first Held out by task: the test asks are different situations from the training asks, not different rollouts of the same ones. Roll the current agent on it three times so the eval's own re-run spread is on record; that spread is the `noise_floor` a gain has to clear ("Evaluation": one evaluation is a draw). ```python K = 4 # rollouts per ask SIM = dict( tools=TOOLS, system_prompt=POLICY, simulator=False, mode="rl", repeats=K, repeat_policy="fixed", reproducible=True, seed=0, concurrency=1, ) test = wai.simulate( before, seeds=TEST_ASKS, situations=len(TEST_ASKS), budget=len(TEST_ASKS) * K, **SIM ) test_tasks = {r["scenario_id"] for r in test.trajectories} base = wai.simulate(before, tasks=test, ru