sft-from-traceslisted
Install: claude install-skill whilehq/whileai-sdk
# SFT from traces
Hold out a test by task first, sample more attempts around the failures
the traces show, keep what a program judge approves, export a TRL file
with loss masks, and score the fine-tuned agent on the same frozen test.
`check.py` runs every block below offline. Its setup defines the names the
blocks use: `TRACES_PATH` (a JSONL the fixture writes, 40 rows from a
scripted refund agent), `TOOLS` and `POLICY`, `before_agent` (the agent in
production, an `agent(message) -> {"steps", "final_text"}` callable),
`after_agent` (the stand-in for the fine-tuned model), `refund_judge` and
`length_judge` (judges that are programs), `fake` (a recording platform
transport), `K = 4`, `TEST_VERSION = "heldout-v1"` and `OUT_DIR`.
## 1. Read the traces
**Load and report.** `load_traces` normalizes the supported shapes;
`trace_report` says what the traces carry and where generation will aim.
```python
traces = wai.load_traces(str(TRACES_PATH))
report = wai.trace_report(traces, tools=TOOLS, policy=POLICY)
print(wai.format_trace_report(report))
```
## 2. Freeze the held-out test first
**Hold out by task, never by row.** A task is a situation; two rows of the
same ask land on the same side. Traces with no `task_id` need one per
distinct ask before this step.
```python
tasks = sorted({t["task_id"] for t in traces})
held = set(tasks[::3])
heldout_traces = [t for t in traces if t["task_id"] in held]
train_traces = [t for t in traces if t["task_id"] not in held]
```
**Roll the sa