a-second-model-family-before-a-fifth-sample

# Buy the second model family before the fifth sample of the first When a released checkpoint does the predicting rather than a model you fit, the largest single determinant of the score is which checkpoints are on disk *and how unlike each other they are*. This skill is about the set, not about picking the best single member of it. That decision usually gets made in the first ten minutes, from one search, before anything has been measured — and then it is never revisited, because every later hour goes on prompts, decoding and voting *inside* the set it created. Measured on one run of this shape: two checkpoints were on disk eight minutes in, and ten hours later they were still the whole pool — a 1.5B and a 7B of one released family. A second agent on the same task with the same clock kept adding checkpoints for two and a half hours, finished with six across five unrelated lineages, and beat it by 4.7 accuracy points. ## Another sample of the same checkpoint is not another candidate Decodes of one checkpoint at different seeds and temperatures fail on the same rows, so they add votes without adding information. On that run's own 200 labelled held-out rows, six decodes of its two checkpoints scored between 144 and 192; the best of all 63 subset votes was 194; the oracle over all six — count a row correct if *any* decode gets it — was 196. What that pool was actually worth is visible only afterwards, from the withheld labels, and it is the reason this skill exists: on the ...