← ClaudeAtlas

backtest-overfittinglisted

Decide whether an edge that passed every mechanical check is still just the best of N tries. TRIGGER - PBO, probability of backtest overfitting, CSCV, combinatorially symmetric cross validation, logit of the out-of-sample rank; minimum backtest length, MinBTL, "how much history do I need", "how many parameter sets can I try on N years of data"; a grid search, Optuna or AutoML picked a winner and it decayed; "in-sample Sharpe 2, live Sharpe 0", "the best parameter set stopped working", "is this peak on my parameter surface real"; pypbo, RiskLabAI CSCV. SKIP for the deflated and probabilistic Sharpe ratios, the trial ledger and SPA/StepM/MCS against a benchmark (backtest-validation); for family-wise error and false-discovery control over a whole research programme (multiple-testing-ledger); for the purged and combinatorial CV splitters themselves (lib-purgedcv); for the mechanical leakage gates that come first (research-integrity-guards); and for regime coverage of the test window (regime-detection).
howard-lynn-ye/fin-skills · ★ 1 · Testing & QA · score 77
Install: claude install-skill howard-lynn-ye/fin-skills
# Backtest overfitting **Seven guards in this repo ask whether a backtest cheated. This one asks the question that is still open after they all pass: the backtest is clean, and it is *the maximum of a search*.** Two numbers answer it. **PBO** asks how often the in-sample winner lands in the bottom half out of sample — 0.5 means the winner is a coin flip. **MinBTL** asks how many years of data a claimed Sharpe needs before the honest trial count stops explaining it on its own. Every number below is printed by `scripts/overfitting.py` (numpy + scipy, seeds fixed and printed, **24 s**). Nothing here is quoted from a paper without being reproduced. ## 1. CSCV, exactly as defined ✅ Source: Bailey, Borwein, López de Prado & Zhu, *The Probability of Backtest Overfitting*, Journal of Computational Finance 20(4), 2016 — read from the authors' hosted preprint (`davidhbailey.com/dhbpapers/backtest-prob.pdf`; the published JCF text is paywalled and was not reached). Algorithm 2.3: 1. Build a **T x N** performance matrix — one column per configuration you tried, rows aligned on the same calendar. 2. Split the **rows** into an even number **S** of disjoint contiguous blocks. 3. Form all **C(S, S/2)** ways of choosing S/2 blocks as in-sample; the complement is out-of-sample. Both halves are the same size — that is what "symmetric" means, and it is why IS and OS performance are directly comparable. 4. `n*` = the configuration with the best IS performance. Find its **rank** amo