cv-lb-debugginglisted
Install: claude install-skill TranBaDat2607/claude-code-kaggle-grandmaster
# CV ↔ LB Debugging
## Symptom → likely causes
| Symptom | Likely causes (check in order) |
|---|---|
| CV ≫ LB (CV too optimistic) | group leakage (same entity in train/val), temporal leakage, target-derived features fit outside folds, duplicates across folds, preprocessing fit on all data, post-processing tuned on reported OOF |
| CV ≪ LB | validation harder than test (e.g. grouped CV but test shares groups with train → test has "seen" entities; could exploit), noisy small public LB, val includes noisy labels absent from test |
| CV improves, LB flat/down | small/noisy public split, change exploits a train-only artefact, distribution shift on that feature, inference pipeline differs |
| LB much worse than any CV fold | submission bug: id misalignment, sorting, wrong column, probabilities vs labels, wrong transform inverse (expm1), different preprocessing at inference |
| Huge sudden CV jump | leak — assume guilty until proven otherwise |
| High fold variance | small data / noisy metric; use more folds, repeated CV, seeds |
## Procedure
1. **Re-score the submitted file offline**: run the inference code on *training* data (or a
validation fold) and check it reproduces OOF scores — catches inference/training mismatch.
2. **Sanity-check the submission**: `python -m kgkit validate`; compare prediction distribution
(mean, quantiles) of test vs OOF; correlation of the new submission with the previous best.
3. **Leakage audit** (dispatch `validation-auditor` agent): list