evaluating-a-tuned-modellisted

# Evaluating a tuned model ## Compare it against its own base, not against a vague memory of "before" This is the argument the rest of this skill exists to make operational, so it comes first: a fine-tune has to be measured against the base model it started from, run on the same probes, under the same decoding settings. Most claimed improvements do not survive that comparison. A handful of prompts that felt better after training, a training loss curve that went down, or a memory of how bad the base model used to be are not a measurement, they are an impression. Skip the base comparison and ship on an impression, and the first time someone asks "how do you know it's better" there is no answer. The base model is also the only fair opponent. Comparing a fine-tune to a much larger frontier model and losing tells you nothing about whether training worked, because you already knew a small model would lose to a frontier model before training even started. The question training answers is narrower and more useful: did this task-specific training move the needle from where this exact base model started. A frontier comparison is still worth running, but as a second, optional data point about where the tuned model sits in the wider market, not as the test of whether training did anything. ## The shape of a defensible eval Four pieces, produced in this order: 1. **`eval/probes.jsonl`**, a set of held-out inputs the model has to handle, one JSON object per line, each shaped `{