multinode-traininglisted

# Multinode Training Treat multi-node training as a distributed system with three separate contracts: 1. the computation contract: parallel dimensions, batch semantics, collectives, and optimizer equivalence; 2. the control-plane contract: node identity, rendezvous, process ownership, barriers, failure propagation, and recovery; 3. the dependency contract: code, model assets, data, local services, logs, checkpoints, and external trackers. A successful `torchrun` command is not proof that all three contracts are correct. ## Load only the needed references - Read [topology-and-semantics.md](references/topology-and-semantics.md) before changing world size, parallelism, batching, gradient accumulation, sampling groups, or checkpoint code. - Read [reliable-launch.md](references/reliable-launch.md) before writing a launcher, supervisor, remote-agent prompt, asset preparation flow, or recovery policy. - Read [diagnostics.md](references/diagnostics.md) when a job hangs, one node diverges, throughput regresses, metrics disagree, or startup does not reach training. - Read [source-notes.md](references/source-notes.md) when refreshing the skill or checking which rules come from PyTorch, Accelerate, DeepSpeed, Megatron-LM, Hugging Face Hub, or NCCL. ## Establish a frozen run contract Before editing code or allocating GPUs, write down and mechanically validate: - experiment ID, attempt, launch nonce, exact commit, repository cleanliness, and config hash; - ordered host list, expect