← ClaudeAtlas

validation-strategylisted

Designing a cross-validation scheme that tracks the private leaderboard — choosing KFold/Stratified/Group/StratifiedGroup/multilabel/time-series/purged splits, adversarial validation, CV-LB correlation, noise estimation, and leakage prevention. Use when setting up CV, when CV and LB disagree, or before trusting any score.
TranBaDat2607/claude-code-kaggle-grandmaster · ★ 2 · AI & Automation · score 75
Install: claude install-skill TranBaDat2607/claude-code-kaggle-grandmaster
# Validation Strategy **Rule:** the validation split must be generated the same way the test split was generated. Find out how train/test were separated (random rows? by patient/user/site? by time? by geography?) and imitate it. ## 1. Decision table | How test differs from train | Scheme | kgkit | |---|---|---| | Random rows, iid | StratifiedKFold (classification) / KFold or binned-stratified (regression) | `assign_folds(strategy="stratified")` | | Unseen groups (patients, users, images of same object, sessions, sites) | GroupKFold / StratifiedGroupKFold | `strategy="stratified_group", group=...` | | Multilabel targets | iterative stratification | `target=[cols...]` | | Future period | forward-chaining time split with gap ≥ horizon | `time_series_splits(t, gap=h)` | | Overlapping-window targets (finance) | purged K-fold with embargo | `purged_kfold_splits(t, embargo=k)` | | Unseen groups *and* future | time split, with groups in validation unseen in train | combine both | | Strong covariate shift | adversarial validation → test-like holdout or weights | `kgkit.adversarial` | Hidden group structure is the #1 source of CV/LB gaps: near-duplicate images, same customer across rows, same text template, augmentations of one source, multiple rows per patient-visit. Look for them (hash images, cluster texts, group by obvious IDs) and group. ## 2. Mechanics - `python -m kgkit folds data/train.csv --target y --group g --id-col id --out data/folds.csv` then **commit folds.csv**