← ClaudeAtlas

a-comparison-you-never-run-defaults-to-your-preferencelisted

Use at hypothesis drafting, study design and implementation when the task could plausibly be attacked by more than one family of method - hand-built features fed to a fitted model, a network trained on the raw structure or sequence, a pretrained backbone, retrieval - and your hypothesis list mostly compares variants inside one of them. Covers separating the hypotheses that would change what you build from the ones that would change an argument, requiring code on both sides of a family claim, running the comparison at a budget you actually have and reading each side's slope rather than its level, and demoting a comparison you will not run into a priced assumption.
tangxiangru/AutoR · ★ 805 · AI & Automation · score 79
Install: claude install-skill tangxiangru/AutoR
# The hypothesis that chooses the method family runs first, at full budget Most hypotheses a run registers are internal to a method it has already picked: which feature block, which parameterisation, which form of the correction term. One or two decide something larger — which *family* the run is in. Those are the only ones whose refutation would change what gets built, and they are reliably the ones that do not get resolved, because they are the most expensive and they sit at the end of the queue. An unrun comparison is not neutral. **It resolves, silently, in favour of whatever you were already doing.** ## What the failure looks like from inside It does not look like ignorance, and it does not look like laziness. It looks like careful measurement, taken at the wrong size. One run's literature stage tabulated the published ladder honestly: the best number from the family it did not choose was 0.022, the best from the family it did was 0.044 — two to one. That finding entered the hypothesis document once, as H6: *at a matched CPU budget the descriptor route reaches a lower fold error than a message-passing network trained from the same coordinates.* The words naming that rival family then occur **zero times in the 16 KB study design**. The run did not ignore it. It measured it twice: - At **09:05** a cost pilot timed a real step of the rival family: 106.5 s per epoch on 100,831 molecules, on the three cores it had. Six hours of clock remained — about **200 epochs**