debuglisted

# Debug: evidence-before-action investigation The most expensive class of mistake in ML debugging is asserting a cause based on plausibility, then attempting a "fix" that masks the real problem. This skill enforces the discipline of probe → hypothesis → smoke → controls → claim, in that order. The agentic Stop hook routes here from `reason` when an assistant claims a cause without backing tool output. ## When to run The user just said any of: - "why is X failing / diverging / NaN / OOM / hung / slow / crashed" - "the loss is going up", "metrics look weird", "GPU util is 0" - "debug this", "diagnose", "troubleshoot", "investigate this run" - pasted a log excerpt asking what's wrong ## Five-step protocol ### Step 1: cheap probes Before forming any hypothesis, gather the cheap evidence. None of these cost more than a few seconds: **Process state:** ```bash ps aux | grep -E '(python|train|torchrun|accelerate)' | grep -v grep ``` Is the process still running? Zombie? Defunct? Multiple instances? **Kernel / system events:** ```bash dmesg | tail -100 # OOM kills, hardware errors, NFS errors journalctl -xe --since "1 hour ago" | tail -50 ``` **GPU state:** ```bash nvidia-smi nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu --format=csv ``` Is the GPU even being used? Idle GPU during "training" means the process is blocked on data loading or has died. **Disk / filesystem:** ```bash df -h /path/to/run-dir du -sh /path/to/run-dir/* ``` Out of disk? C