deep-learning-traininglisted
Install: claude install-skill TranBaDat2607/claude-code-kaggle-grandmaster
# Deep Learning Training for Competitions
## Default recipe
- Optimiser AdamW (β 0.9/0.999, wd 1e-2; no wd on norms/biases), cosine decay with linear warmup
(3–10% of steps), peak LR found by a short LR range test or known defaults (CNN fine-tune
1e-4–5e-4, transformers 1e-5–3e-5, heads ×10).
- Mixed precision: `torch.autocast` bf16 on Ampere+ (A100/L4/H100), fp16 + GradScaler on T4/P100.
- Gradient clipping (max norm 1.0) for transformers/RNNs.
- **EMA of weights** (decay 0.999–0.9999) — cheap, frequently +, evaluate the EMA model.
- Checkpoint the best-by-metric (not by loss) per fold; also keep last for SWA if used.
- Evaluate more often than once per epoch for short schedules.
## Speed (more experiments per GPU-hour)
Every training loop starts with these defaults. Load the `compute-efficiency` skill when writing or changing a
loop: it has the measurement, the per-domain how-to and what runs on which GPU.
- **Measure first**: the templates' `--bench` times the data loader alone, the training step alone and the real
loop (`python -m kgkit bench --run "<cmd>"` for any script). Input-bound: fix the pipeline. Compute-bound: less
compute (AMP, compile, smaller inputs).
- AMP (above), `channels_last` for CNNs, `cudnn.benchmark = True` for fixed input sizes, fused AdamW, SDPA attention.
- Inputs cached once at training size (images as uint8 npy/PNG, text as token ids). `num_workers` = CPU cores ÷
processes (two folds on Kaggle's 4 cores: 2 each), `pin_memory`, `per