run-a-control-before-you-trust-the-numberlisted

Use when a harness, script or query has just produced a number you are about to act on — put in a PR body, use to rank options, or use to decide a change is unnecessary. A broken measurement almost never raises an error; it returns a plausible answer. So before trusting the instrument, run a probe whose correct output you already know and that the broken state CANNOT produce. Also check the three silent distortions: a suppressed error read as an absence, a window of N items read as a window of time, and a cross-product reported as coverage. Trigger terms: benchmark, A/B, measured, the numbers say, 0 hits, no difference, nothing found, last N commits, N x M, ranked by, harness, probe, sweep, no measurable cost, tests pass so it works.
serhiy-bzhezytskyy/contrib-receipts · ★ 1 · Code & Development · score 75

Install: claude install-skill serhiy-bzhezytskyy/contrib-receipts

# Run a control before you trust the number ## Purpose A number is the most persuasive thing you can hand a reviewer, and a broken instrument is the hardest failure to notice, because **it does not fail**. It returns something shaped like an answer — often the answer you expected, which is worse. `0.000 vs 0.000` reads as "no difference". `0 hits` reads as "not present". `+22%` reads as a cost you had predicted. None of them announces that the harness never loaded, the grep could not see the line, or the two medians came from different sessions. This is distinct from asking whether a number clears its noise floor ([state-the-noise-floor](../state-the-noise-floor/SKILL.md)). That question assumes the instrument works. This one asks the prior question: **is this measuring what I think it is measuring?** A number can sit far above its noise floor and still be an artifact. The remedy is cheap and specific: a control whose correct output you already know, chosen so that the broken state and the working state give *different* answers. ## When to use - A harness, script, benchmark or query has produced a figure you intend to publish, rank by, or use to justify not making a change. - The result is reassuring — a tie, a zero, "nothing found", "no measurable difference". Reassuring output deserves *more* suspicion, not less. - You are comparing two subjects and the comparison was assembled from separate runs. - You are about to report a count as coverage ("34 × 40 = 1360 comp