← ClaudeAtlas

deep-learning-traininglisted

Competition-grade deep learning training practice in PyTorch — schedules, optimisers, mixed precision, EMA/SWA, gradient accumulation and checkpointing, multi-GPU on Kaggle (2xT4), data loading speed, reproducibility, debugging loss curves, and making the most of limited GPU hours. Use when writing or debugging any neural network training loop for a competition.
TranBaDat2607/claude-code-kaggle-grandmaster · ★ 2 · AI & Automation · score 75
Install: claude install-skill TranBaDat2607/claude-code-kaggle-grandmaster
# Deep Learning Training for Competitions ## Default recipe - Optimiser AdamW (β 0.9/0.999, wd 1e-2; no wd on norms/biases), cosine decay with linear warmup (3–10% of steps), peak LR found by a short LR range test or known defaults (CNN fine-tune 1e-4–5e-4, transformers 1e-5–3e-5, heads ×10). - Mixed precision: `torch.autocast` bf16 on Ampere+ (A100/L4/H100), fp16 + GradScaler on T4/P100. - Gradient clipping (max norm 1.0) for transformers/RNNs. - **EMA of weights** (decay 0.999–0.9999) — cheap, frequently +, evaluate the EMA model. - Checkpoint the best-by-metric (not by loss) per fold; also keep last for SWA if used. - Evaluate more often than once per epoch for short schedules. ## Speed (more experiments per GPU-hour) Every training loop starts with these defaults. Load the `compute-efficiency` skill when writing or changing a loop: it has the measurement, the per-domain how-to and what runs on which GPU. - **Measure first**: the templates' `--bench` times the data loader alone, the training step alone and the real loop (`python -m kgkit bench --run "<cmd>"` for any script). Input-bound: fix the pipeline. Compute-bound: less compute (AMP, compile, smaller inputs). - AMP (above), `channels_last` for CNNs, `cudnn.benchmark = True` for fixed input sizes, fused AdamW, SDPA attention. - Inputs cached once at training size (images as uint8 npy/PNG, text as token ids). `num_workers` = CPU cores ÷ processes (two folds on Kaggle's 4 cores: 2 each), `pin_memory`, `per