validation-strategylisted
Install: claude install-skill TranBaDat2607/claude-code-kaggle-grandmaster
# Validation Strategy
**Rule:** the validation split must be generated the same way the test split was generated.
Find out how train/test were separated (random rows? by patient/user/site? by time? by
geography?) and imitate it.
## 1. Decision table
| How test differs from train | Scheme | kgkit |
|---|---|---|
| Random rows, iid | StratifiedKFold (classification) / KFold or binned-stratified (regression) | `assign_folds(strategy="stratified")` |
| Unseen groups (patients, users, images of same object, sessions, sites) | GroupKFold / StratifiedGroupKFold | `strategy="stratified_group", group=...` |
| Multilabel targets | iterative stratification | `target=[cols...]` |
| Future period | forward-chaining time split with gap ≥ horizon | `time_series_splits(t, gap=h)` |
| Overlapping-window targets (finance) | purged K-fold with embargo | `purged_kfold_splits(t, embargo=k)` |
| Unseen groups *and* future | time split, with groups in validation unseen in train | combine both |
| Strong covariate shift | adversarial validation → test-like holdout or weights | `kgkit.adversarial` |
Hidden group structure is the #1 source of CV/LB gaps: near-duplicate images, same customer
across rows, same text template, augmentations of one source, multiple rows per
patient-visit. Look for them (hash images, cluster texts, group by obvious IDs) and group.
## 2. Mechanics
- `python -m kgkit folds data/train.csv --target y --group g --id-col id --out data/folds.csv`
then **commit folds.csv**