backtest-validationlisted

# Backtest Validation A backtest is a claim, not a result. In-sample performance is the *cheapest* number in quant research: it rises with every parameter you tune and every variant you discard. This skill converts "the backtest looks great" into a defensible verdict — or an honest rejection. ## The four ways a backtest lies 1. **Selection bias.** You tried N variants and reported the best. The maximum of N noise strategies has a positive expected Sharpe that grows with N. Ask the user how many variants were tried; disclose that number to the Deflated Sharpe Ratio. Undisclosed trials are the most common lie by omission. 2. **Cost blindness.** Gross returns ignore what turnover costs. An edge that dies at 10 bps per unit of turnover was never an edge — it was a liquidity donation you had not made yet. 3. **In-sample memorization.** Parameters fitted on the full history describe the past, not the future. A chronological in-sample / out-of-sample split is the minimum; walk-forward is better when the series is long enough. 4. **Fragility.** One draw of history is one draw. If a stationary block bootstrap says p(Sharpe ≤ 0) is 10%, the "edge" is statistically indistinguishable from luck regardless of how the equity curve looks. Look-ahead and revision leakage are the *fifth* way — that is `point-in-time-research`'s jurisdiction. When the input data has not passed PIT discipline, run that skill first; a gauntlet on contaminated data validates nothing.