← ClaudeAtlas

mkl-debug-ml-trainingsolid

Investigate training failures, NaNs, missing gradients, misleading losses, and non-reproducible runs in PyTorch, Lightning, or TensorFlow/Keras. Use for a concrete training bug or regression, not an open-ended architecture or hyperparameter search.
00200200/maintainer-skills-lab · ★ 35 · AI & Automation · score 82
Install: claude install-skill 00200200/maintainer-skills-lab
# Debug a training run Turn the reported symptom into an observable failure on a bounded input. Preserve the user's framework, backend, and intended training semantics. A CPU or eager-mode reproduction can help isolate a bug, but does not establish what happens on the original accelerator, compiled graph, or distributed setup. ## Establish the failing boundary Record the command, revision, Python and framework versions, imported package paths, device, dtype, precision, and execution mode. For a resume or reproducibility issue, include the data order, seed handling, checkpoint, optimizer, scheduler, and gradient-scaler state that affect the failing step. Classify dependency/import failures separately from a reproduced training bug. Reduce to a fixed batch while retaining the relevant shapes, label semantics, masks, and value ranges. Inspect the tensors actually entering the loss. Compare their shapes to that loss's contract: class indices, probability targets, and scalar regression targets have different requirements. Do not apply a blanket reshape or squeeze to silence an error. For example, scalar predictions of shape `[N, 1]` minus labels of shape `[N]` produce an `[N, N]` residual through broadcasting. With predictions `[[1], [3]]` and labels `[1, 3]`, the raw mean squared residual is `2` despite perfect per-example predictions. Aligning these scalar labels to `[N, 1]` makes it `0`. This is a bug in that loss expression; built-in losses may validate or normalize shape