← ClaudeAtlas

cv-lb-debugginglisted

Systematic diagnosis when cross-validation and leaderboard disagree, when a score is suspiciously good or bad, or when an improvement in CV does not transfer to LB — leakage hunting, distribution shift analysis, inference/training mismatch, metric bugs, and public LB noise estimation. Use when CV and LB diverge or a result looks wrong.
TranBaDat2607/claude-code-kaggle-grandmaster · ★ 2 · AI & Automation · score 75
Install: claude install-skill TranBaDat2607/claude-code-kaggle-grandmaster
# CV ↔ LB Debugging ## Symptom → likely causes | Symptom | Likely causes (check in order) | |---|---| | CV ≫ LB (CV too optimistic) | group leakage (same entity in train/val), temporal leakage, target-derived features fit outside folds, duplicates across folds, preprocessing fit on all data, post-processing tuned on reported OOF | | CV ≪ LB | validation harder than test (e.g. grouped CV but test shares groups with train → test has "seen" entities; could exploit), noisy small public LB, val includes noisy labels absent from test | | CV improves, LB flat/down | small/noisy public split, change exploits a train-only artefact, distribution shift on that feature, inference pipeline differs | | LB much worse than any CV fold | submission bug: id misalignment, sorting, wrong column, probabilities vs labels, wrong transform inverse (expm1), different preprocessing at inference | | Huge sudden CV jump | leak — assume guilty until proven otherwise | | High fold variance | small data / noisy metric; use more folds, repeated CV, seeds | ## Procedure 1. **Re-score the submitted file offline**: run the inference code on *training* data (or a validation fold) and check it reproduces OOF scores — catches inference/training mismatch. 2. **Sanity-check the submission**: `python -m kgkit validate`; compare prediction distribution (mean, quantiles) of test vs OOF; correlation of the new submission with the previous best. 3. **Leakage audit** (dispatch `validation-auditor` agent): list