dpo-pairslisted
Install: claude install-skill whilehq/whileai-sdk
# DPO pairs from graded rollouts
DPO trains on pairs: same ask, one reply preferred, no reward model in the
loop ("Direct Alignment"). Where the same ask passed once and failed once, the
contrast is the signal.
Names from `check.py` used below: `TOOLS` and `POLICY` (a refund agent's
tools and rules), `TRAIN_ASKS` and `TEST_ASKS` (two disjoint ask lists),
`before` (the agent you have; a scripted bot that refunds before looking the
order up on half its rollouts), `after` (stands in for the model trained on the
pairs), `refund_judge` (the policy as a program), `train_reward` (one rule of
it, the training signal), `length_judge` (an untrained behavior), `fake` (a
recording transport), `OUT` (an output folder).
## 1. Freeze the held-out test first
Held out by task: the test asks are different situations from the training
asks, not different rollouts of the same ones. Roll the current agent on it
three times so the eval's own re-run spread is on record; that spread is the
`noise_floor` a gain has to clear ("Evaluation": one evaluation is a draw).
```python
K = 4 # rollouts per ask
SIM = dict(
tools=TOOLS,
system_prompt=POLICY,
simulator=False,
mode="rl",
repeats=K,
repeat_policy="fixed",
reproducible=True,
seed=0,
concurrency=1,
)
test = wai.simulate(
before, seeds=TEST_ASKS, situations=len(TEST_ASKS), budget=len(TEST_ASKS) * K, **SIM
)
test_tasks = {r["scenario_id"] for r in test.trajectories}
base = wai.simulate(before, tasks=test, ru