calibrate-the-level-on-the-window-you-cannot-score

# Your bias check was run somewhere else Two sentences sound identical and are not: - *my predictions are unbiased on the folds I can score* - *my predictions are unbiased on the rows that will be graded* The first is a measurement. The second is an extrapolation from it, and it is only safe when the graded rows are exchangeable with the folds. When the graded split is a **later period, a different site, a held-out batch, or a horizon that starts where the supplied history ends**, they are not exchangeable — that is the whole reason the split was cut that way — and the extrapolation is usually the largest un-audited assumption in the run. The check you can run is the first sentence. The number you are paid on is the second. This skill is about closing that distance without labels. ## Why the level, specifically For any error sensitive to scale — absolute error, squared error, a scaled or percentage error, anything divided by a per-unit denominator — a single global multiplier on all your predictions is: - **one degree of freedom**, so it cannot meaningfully overfit; - **flat near its optimum**, so being roughly right captures nearly all of it; - **the only correction you can apply to a region you cannot score at all**, because it needs no per-row information. That makes it the highest-leverage-per-minute knob you own late in a run, and the one most reliably declared finished early — because the check that closed it was cheap and returned a comforting answer. ## 1. ...